-- Logs begin at Tue 2022-08-30 04:35:15 UTC, end at Tue 2022-08-30 05:33:24 UTC. -- Aug 30 04:58:05.312924 dv-cinder-853910-4-iscsi systemd[1]: Starting Devstack devstack@keystone.service... Aug 30 04:58:05.321850 dv-cinder-853910-4-iscsi devstack@keystone.service[81618]: [uWSGI] getting INI configuration from /etc/keystone/keystone-uwsgi-public.ini Aug 30 04:58:05.324367 dv-cinder-853910-4-iscsi devstack@keystone.service[81618]: *** Starting uWSGI 2.0.18-debian (64bit) on [Tue Aug 30 04:58:05 2022] *** Aug 30 04:58:05.324367 dv-cinder-853910-4-iscsi devstack@keystone.service[81618]: compiled with version: 10.0.1 20200405 (experimental) [master revision 0be9efad938:fcb98e4978a:705510a708d3642c9c962beb663c476167e4e8a4] on 11 April 2020 11:15:55 Aug 30 04:58:05.324367 dv-cinder-853910-4-iscsi devstack@keystone.service[81618]: os: Linux-5.4.0-72-generic #80-Ubuntu SMP Mon Apr 12 17:35:00 UTC 2021 Aug 30 04:58:05.324367 dv-cinder-853910-4-iscsi devstack@keystone.service[81618]: nodename: dv-cinder-853910-4-iscsi Aug 30 04:58:05.324367 dv-cinder-853910-4-iscsi devstack@keystone.service[81618]: machine: x86_64 Aug 30 04:58:05.324367 dv-cinder-853910-4-iscsi devstack@keystone.service[81618]: clock source: unix Aug 30 04:58:05.324367 dv-cinder-853910-4-iscsi devstack@keystone.service[81618]: pcre jit disabled Aug 30 04:58:05.324367 dv-cinder-853910-4-iscsi devstack@keystone.service[81618]: detected number of CPU cores: 4 Aug 30 04:58:05.324367 dv-cinder-853910-4-iscsi devstack@keystone.service[81618]: current working directory: / Aug 30 04:58:05.324367 dv-cinder-853910-4-iscsi devstack@keystone.service[81618]: detected binary path: /usr/bin/uwsgi-core Aug 30 04:58:05.324367 dv-cinder-853910-4-iscsi devstack@keystone.service[81618]: your processes number limit is 47680 Aug 30 04:58:05.324367 dv-cinder-853910-4-iscsi devstack@keystone.service[81618]: your memory page size is 4096 bytes Aug 30 04:58:05.324367 dv-cinder-853910-4-iscsi devstack@keystone.service[81618]: detected max file descriptor number: 2048 Aug 30 04:58:05.325044 dv-cinder-853910-4-iscsi devstack@keystone.service[81618]: lock engine: pthread robust mutexes Aug 30 04:58:05.325044 dv-cinder-853910-4-iscsi devstack@keystone.service[81618]: thunder lock: enabled Aug 30 04:58:05.325044 dv-cinder-853910-4-iscsi devstack@keystone.service[81618]: uwsgi socket 0 bound to UNIX address /var/run/uwsgi/keystone-wsgi-public.socket fd 4 Aug 30 04:58:05.325044 dv-cinder-853910-4-iscsi devstack@keystone.service[81618]: Python version: 3.8.10 (default, Jun 22 2022, 20:18:18) [GCC 9.4.0] Aug 30 04:58:05.363181 dv-cinder-853910-4-iscsi devstack@keystone.service[81618]: Python main interpreter initialized at 0x555ec70ab9c0 Aug 30 04:58:05.363181 dv-cinder-853910-4-iscsi devstack@keystone.service[81618]: python threads support enabled Aug 30 04:58:05.363181 dv-cinder-853910-4-iscsi devstack@keystone.service[81618]: your server socket listen backlog is limited to 100 connections Aug 30 04:58:05.363181 dv-cinder-853910-4-iscsi devstack@keystone.service[81618]: your mercy for graceful operations on workers is 90 seconds Aug 30 04:58:05.363548 dv-cinder-853910-4-iscsi devstack@keystone.service[81618]: mapped 403077 bytes (393 KB) for 2 cores Aug 30 04:58:05.363638 dv-cinder-853910-4-iscsi devstack@keystone.service[81618]: *** Operational MODE: preforking *** Aug 30 04:58:05.363879 dv-cinder-853910-4-iscsi devstack@keystone.service[81618]: *** uWSGI is running in multiple interpreter mode *** Aug 30 04:58:05.363879 dv-cinder-853910-4-iscsi devstack@keystone.service[81618]: spawned uWSGI master process (pid: 81618) Aug 30 04:58:05.364089 dv-cinder-853910-4-iscsi systemd[1]: Started Devstack devstack@keystone.service. Aug 30 04:58:05.366059 dv-cinder-853910-4-iscsi devstack@keystone.service[81618]: spawned uWSGI worker 1 (pid: 81624, cores: 1) Aug 30 04:58:05.366059 dv-cinder-853910-4-iscsi devstack@keystone.service[81618]: spawned uWSGI worker 2 (pid: 81625, cores: 1) Aug 30 04:58:05.366059 dv-cinder-853910-4-iscsi devstack@keystone.service[81618]: running "unix_signal:15 gracefully_kill_them_all" (master-start)... Aug 30 04:58:09.603630 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Aug 30 04:58:09.603630 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: from cryptography.utils import int_from_bytes Aug 30 04:58:09.604637 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Aug 30 04:58:09.604637 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: from cryptography.utils import int_from_bytes Aug 30 04:58:09.606104 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Aug 30 04:58:09.606104 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: from cryptography.utils import int_from_bytes Aug 30 04:58:09.606528 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Aug 30 04:58:09.606528 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: from cryptography.utils import int_from_bytes Aug 30 04:58:09.653731 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Aug 30 04:58:09.653920 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Aug 30 04:58:09.654110 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] command line args: [] {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Aug 30 04:58:09.654377 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Aug 30 04:58:09.654551 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] ================================================================================ {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Aug 30 04:58:09.654803 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] admin_token = **** {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.655069 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] config_dir = [] {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.655254 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.655433 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] config_source = [] {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.655607 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] debug = True {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.655802 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.656297 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.656400 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.656535 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Aug 30 04:58:09.656622 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.656726 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Aug 30 04:58:09.656806 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.656911 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] command line args: [] {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Aug 30 04:58:09.657017 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] list_limit = None {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.657183 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] log_config_append = None {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.657287 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Aug 30 04:58:09.657386 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] ================================================================================ {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Aug 30 04:58:09.657480 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.657587 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] log_dir = None {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.657681 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] admin_token = **** {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.657773 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] log_file = None {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.657870 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] config_dir = [] {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.657971 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.658072 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.658158 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.658237 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] config_source = [] {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.658361 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] debug = True {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.658441 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.658553 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.658665 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.658740 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.658887 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.658969 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.659047 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.659171 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.659292 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.659368 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.659490 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.659600 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.659677 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] list_limit = None {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.659801 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] log_config_append = None {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.659880 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.659956 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.660070 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.660150 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] log_dir = None {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.660256 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.660341 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] log_file = None {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.660497 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.660573 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.660737 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.660824 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.660995 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.661077 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.661244 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.661429 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.661512 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.77/identity {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.661608 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.661696 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] publish_errors = False {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.661853 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.661932 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.662077 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.662193 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.662284 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.662361 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.662476 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.662556 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.662695 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.662772 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.662944 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] standard_threads = False {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.663029 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.663174 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.663250 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.663412 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.663487 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.663585 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.663733 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.663827 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] use_journal = False {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.663916 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] use_json = False {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.664085 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] use_stderr = False {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.664217 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.77/identity {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.664295 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] use_syslog = False {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.664403 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] publish_errors = False {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.664491 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.664638 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.664737 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.664870 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.664947 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.665050 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.665181 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.665298 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.665375 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.665534 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.665612 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.665797 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] standard_threads = False {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.665881 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.666032 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.666109 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.666275 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.666357 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.666455 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.666540 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.666617 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] use_journal = False {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.666763 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.666870 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] use_json = False {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.666993 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] use_stderr = False {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.667070 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.667171 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] use_syslog = False {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.667250 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.667406 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:58:09.667489 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.667665 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.667742 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.667906 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.667984 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.668155 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.668234 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.668386 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.668463 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.668635 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.668712 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.668875 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.668953 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.669113 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.669190 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.669346 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.669423 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.669591 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.669672 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.669842 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.669920 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.670088 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.670165 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.670324 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.670400 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.670549 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.670624 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.670773 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.670867 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.671013 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.671101 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.671245 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.671330 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.671486 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.671569 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.671732 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.671815 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.671959 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.672046 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.672185 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.672267 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.672424 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.672506 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.672664 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.672748 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.672896 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.672979 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.673137 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.673219 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.673363 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.673448 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.673593 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.673680 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.673822 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.673898 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.674047 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.674132 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.674270 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.674347 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.674494 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.674576 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.674718 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.674804 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.674972 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.675050 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.675212 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.675289 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.675438 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.675523 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.675668 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.675745 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.675891 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.675967 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.676122 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.676274 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.676347 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.676417 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.676582 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.676659 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.676806 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.676891 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.677037 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.677113 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.677267 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.677349 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.677498 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.677583 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.677728 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.677805 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.677957 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.678066 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.678181 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.678294 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.678430 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.678532 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.678660 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.678762 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.678914 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.679027 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.679140 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.679271 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.679378 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.679458 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.679614 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.679703 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.679821 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.679895 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.680056 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.680133 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.680288 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.680382 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.680523 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.680599 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.680760 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.680844 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.680998 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.681077 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.681229 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.681801 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.681969 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.682051 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.682204 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.682288 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.682435 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.682513 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.682666 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.682751 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.682920 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.682998 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.683154 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.683236 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.683392 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.683480 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.683625 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.683704 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.683862 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.683944 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.684092 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.684171 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.684323 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.684411 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.684554 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.684634 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.684783 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.685013 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.685089 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.685271 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.685361 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.685508 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.685587 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.685750 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.685827 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.685987 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.686072 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.686221 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.686298 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.686457 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.686534 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.686689 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.686774 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.686943 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.687034 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.687174 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.687254 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.687401 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.687489 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.687628 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.687707 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.687853 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.687931 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.688085 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.688239 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.688314 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.688383 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.688534 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.688612 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.688766 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.688853 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.689007 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.689091 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.689239 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.689316 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.689485 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.689566 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.689731 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.689810 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.689965 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.690042 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.690192 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] auth.external = None {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.690270 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.690418 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.690503 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.690648 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.690731 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.690902 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.690984 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.691136 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.691222 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] auth.password = None {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.691373 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.691451 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] auth.token = None {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.691605 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.691712 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.691838 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.691916 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.692075 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.692161 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.692302 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.692380 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.692544 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.692623 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.692788 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.692868 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.693016 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.693095 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.693250 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] credential.caching = True {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.693328 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.693490 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] auth.external = None {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.693569 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.693721 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.693806 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.693961 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.694038 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.694200 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.694280 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.694437 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.694516 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] auth.password = None {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.694668 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.694746 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] auth.token = None {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.694930 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.695012 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.695171 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.695248 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.695403 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.695481 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.695638 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.695728 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.695886 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.695965 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.696135 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.696215 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.696369 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.696449 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.696606 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] credential.caching = True {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.696684 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.696852 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.697059 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.697653 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] federation.caching = True {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.697735 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.697890 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.697969 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.698134 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.698214 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.698376 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.698456 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.698602 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.698680 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.698873 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.698957 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.699100 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.699181 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.699346 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.699424 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.699578 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.699655 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.699828 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.699908 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.700061 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.700139 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.700303 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.700390 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.700535 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] identity.caching = True {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.700613 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.700777 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.700855 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.701019 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.701106 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] federation.caching = True {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.701253 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.701332 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.701485 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.701563 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.701727 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.701813 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.701954 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.702032 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.702215 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.702301 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.702428 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.702583 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.702658 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.702728 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.702907 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.702988 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.703137 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.703227 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.703365 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.703446 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.703605 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.703683 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.703844 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.703931 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] identity.caching = True {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.704084 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.704163 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.704330 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.704407 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.704568 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.704655 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.704805 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.704883 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.705047 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.705132 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.705278 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.705363 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.705502 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.705579 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.705735 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.705863 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.705961 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.706119 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.706199 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.706355 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.706433 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.706597 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.706676 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.706858 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.706941 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.707091 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.707170 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.707340 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.707418 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.707582 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.707668 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.707816 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.707895 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.708057 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.708136 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.708299 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.708378 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.708525 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.708603 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.708758 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.708836 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.708981 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.709060 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.709211 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.709290 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.709443 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.709520 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.709678 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.709756 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.709911 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.709989 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.710142 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.710219 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.710385 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.710464 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.710614 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.710692 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.710864 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.710956 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.711092 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.711171 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.711334 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.711413 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.711575 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.711661 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.711806 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.711884 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.712038 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.712122 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.712273 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.712359 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.712517 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] ldap.user = None {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.712600 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.712757 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.712836 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.712994 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.713086 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.713226 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.713303 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.713464 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.713542 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.713704 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.713790 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.713942 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.714020 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.714175 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.714253 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.714401 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.714487 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.714629 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.714707 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.714882 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.714963 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.715117 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.715208 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.715384 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.715469 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.715584 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.715662 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.715823 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.716063 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.716145 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.716301 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] ldap.user = None {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.716378 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.716542 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.716618 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.716767 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.716846 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.717009 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.717088 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.717241 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.717319 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.717474 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.717553 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.717703 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.717781 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.717933 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.718012 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.718162 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.718240 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.718400 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.718478 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.718630 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.718706 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.718895 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.718977 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.719120 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.719199 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.719357 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.719444 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.719589 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.719669 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.719820 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.719898 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.720052 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.720130 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.720294 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.720392 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.720525 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.720633 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.720772 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.720850 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.721010 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.721089 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.721264 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.721402 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] resource.caching = True {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.721496 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.721574 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.721714 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.721807 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.721965 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.722065 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.722213 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.722304 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.722451 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.722541 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.722691 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.722780 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.722947 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.723037 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.723193 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.723283 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.723426 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.723520 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.723658 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] role.caching = True {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.723747 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.723894 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.723984 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] role.driver = sql {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.724120 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.724208 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.724366 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.724456 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.724596 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.724698 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.724828 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.724930 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.725076 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] resource.caching = True {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.725246 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.725324 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.725397 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.725538 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.725628 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.725778 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.725877 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.726020 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.726110 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.726245 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.726334 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.726485 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.726583 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.726723 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.726850 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.726977 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.727064 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.727220 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.727322 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.727452 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] role.caching = True {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.727602 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.727691 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] role.driver = sql {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.727834 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.727923 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.728078 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.728169 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.728347 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.728436 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.728592 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.728681 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.728837 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.728927 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.729313 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.729405 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.729572 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.729663 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.729820 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.729911 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.730055 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.730144 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.730278 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.730367 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.730506 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.730595 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.730733 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.730843 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.731002 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.731093 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.731249 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.731338 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.731489 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.731639 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.731729 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.731881 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.731971 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.732130 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.732220 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.732379 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] token.caching = True {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.732466 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.732612 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.732703 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.732854 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.732943 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.733088 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.733175 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.733335 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.733423 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.733578 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.733666 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.733808 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.733898 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.734039 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.734127 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.734273 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.734364 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.734508 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.734642 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.734777 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.734932 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.735064 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.735193 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.735319 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.735444 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.735581 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.735708 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.735834 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.735973 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.736099 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.736241 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.736379 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.736515 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] token.caching = True {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.736642 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.736767 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.736892 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] database.connection = **** {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.737029 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.737157 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.737296 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.737428 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.737556 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.737684 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.737824 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.737953 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.738092 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.738222 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.738350 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.738479 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.738607 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.738734 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.738909 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.739035 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.739163 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.739291 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.739418 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.739543 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.739687 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.739814 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.739939 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.740066 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.740192 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.740317 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.740466 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.740609 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.740753 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.740885 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.741016 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.741146 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.741275 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] database.connection = **** {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.741405 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.741557 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.741689 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.741689 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81624) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Aug 30 04:58:09.741946 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.741946 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.741946 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.741946 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.741946 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.741946 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.742658 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.742658 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.742658 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.743071 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.743071 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.743330 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.743533 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.744312 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.744548 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.744791 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.745027 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:58:09.745197 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81625) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Aug 30 04:58:09.770444 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81624) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:58:09.770667 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81624) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:58:09.770915 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81624) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:58:09.773825 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81625) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:58:09.774060 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81625) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:58:09.774290 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81625) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:58:09.779191 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81624) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:58:09.782746 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81625) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:58:09.807306 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81624) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:58:09.807527 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81624) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:58:09.809874 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81625) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:58:09.810107 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81625) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:58:09.860489 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81624) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:58:09.861288 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81625) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:58:09.908864 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81624) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:58:09.909090 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81624) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:58:09.910442 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81625) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:58:09.910681 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81625) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:58:10.065643 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81624) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:58:10.065918 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81624) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:58:10.066057 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81625) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:58:10.066199 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81624) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:58:10.066338 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81625) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:58:10.066487 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81624) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:58:10.066625 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81625) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:58:10.066767 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81624) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:58:10.066939 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81625) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:58:10.067073 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81624) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:58:10.067216 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81625) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:58:10.067216 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81625) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:58:10.089469 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81624) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:58:10.089698 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81624) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:58:10.089918 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81624) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:58:10.090137 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81624) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:58:10.090278 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81625) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:58:10.090409 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81624) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:58:10.090549 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81625) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:58:10.090677 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81624) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:58:10.090804 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81625) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:58:10.090973 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81624) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:58:10.091106 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81625) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:58:10.091106 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81625) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:58:10.091355 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81625) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:58:10.091484 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81625) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:58:10.101656 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81624) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:58:10.102406 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81625) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:58:10.114991 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81624) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:58:10.115215 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81624) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:58:10.115442 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81624) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:58:10.115598 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81625) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:58:10.115728 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81624) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:58:10.115859 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81625) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:58:10.115990 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81624) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:58:10.116121 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81625) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:58:10.116250 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81624) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:58:10.116390 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81625) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:58:10.116520 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81624) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:58:10.116662 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81625) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:58:10.116662 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81625) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:58:10.116905 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81625) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:58:10.133374 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81624) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.133562 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81625) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.133707 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81624) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Aug 30 04:58:10.133844 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81625) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Aug 30 04:58:10.133976 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81624) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.134104 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81625) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.134244 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81624) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Aug 30 04:58:10.134399 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81625) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Aug 30 04:58:10.134541 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81624) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.134667 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81625) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.134791 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81624) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.134958 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81625) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.135085 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81624) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.135210 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81625) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.135334 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81624) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.135511 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81625) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.145594 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81624) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.145757 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81625) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.145889 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81624) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.146017 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81625) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.146155 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81624) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.146283 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81625) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.146418 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81624) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.146546 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81625) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.153351 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81624) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 04:58:10.153570 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81625) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 04:58:10.156251 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81624) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 04:58:10.156504 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81625) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 04:58:10.156632 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81624) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.156792 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81625) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.156937 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81624) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.157066 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81624) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.157195 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81625) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.157338 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81624) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.157466 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81625) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.157593 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81624) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.157721 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81625) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.157846 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81624) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.158000 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81625) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.158084 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81624) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.158158 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81625) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.158332 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81624) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.158415 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81625) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.158603 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81624) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.158756 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81625) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.158889 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81624) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.159063 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81625) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.159337 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81625) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.182455 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81624) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.183028 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81625) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.183933 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81624) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 04:58:10.184213 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81624) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.184499 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81625) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 04:58:10.184787 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81625) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.188712 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81624) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 04:58:10.188995 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81624) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.189271 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81624) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.189467 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81625) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 04:58:10.189758 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81625) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.190322 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81625) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.195696 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81624) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 04:58:10.195982 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81624) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.196718 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81625) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 04:58:10.197031 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81625) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.199950 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81624) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 04:58:10.200240 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81624) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.200522 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81624) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.200802 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81624) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.201074 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81624) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.201153 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81625) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 04:58:10.201412 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81625) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.201701 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81625) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.201988 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81625) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.202267 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81625) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.202549 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81625) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.202847 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81625) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.202953 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81624) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.203118 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81625) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.203238 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81624) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.203515 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81624) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.218853 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81624) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.219134 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81624) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.220126 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81625) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.220409 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81625) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.222986 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81624) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 04:58:10.224301 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81625) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 04:58:10.226067 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81624) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.226363 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81624) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.227556 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81625) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.227857 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81625) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.231495 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81624) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 04:58:10.232867 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81625) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 04:58:10.234595 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81624) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 04:58:10.236010 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81625) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 04:58:10.237693 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81624) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.237992 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81624) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.238280 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81624) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.238561 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81624) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.238852 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81624) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.239236 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81624) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.239326 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81625) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.239596 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81625) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.239885 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81625) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.240170 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81625) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.240460 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81625) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.240752 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81625) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.255545 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81624) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 04:58:10.255830 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81624) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.256095 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81624) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.256355 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81624) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.257086 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81625) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 04:58:10.257376 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81625) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.257646 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81625) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.257920 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81625) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.263110 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=81624) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.264761 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81624) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.264892 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=81625) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.266298 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81624) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.266439 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81625) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.266572 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81624) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.267942 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81625) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.268224 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81625) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.269217 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81624) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 04:58:10.269499 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81624) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.269789 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81624) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.270081 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81624) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.270367 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81624) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.270900 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81625) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 04:58:10.271200 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81625) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.271604 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81625) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.271901 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81625) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.272192 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81625) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.281628 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81624) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 04:58:10.281913 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81624) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.282193 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81624) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.282470 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81624) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.282746 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81624) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.283041 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81624) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.283333 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81624) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.283415 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81625) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 04:58:10.283657 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81625) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.284040 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81625) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.284329 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81625) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.284620 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81625) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.284906 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81625) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.285192 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81625) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.299119 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81624) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 04:58:10.301390 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81625) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 04:58:10.302111 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81624) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 04:58:10.304328 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81625) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 04:58:10.304992 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81624) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.307019 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81624) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.307788 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81625) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.308488 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81624) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 04:58:10.308767 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81624) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.309056 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81624) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.309259 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81625) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.310878 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81625) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 04:58:10.311168 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81625) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.311464 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81625) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.315617 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81624) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.317121 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81624) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 04:58:10.318082 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81625) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.319586 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81625) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 04:58:10.319936 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81624) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.320238 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81624) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.320519 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81624) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.320789 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81624) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.322634 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81625) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.322953 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81625) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.323247 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81625) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.323540 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81625) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.328697 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81624) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 04:58:10.328984 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81624) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.329261 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81624) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.331376 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81625) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 04:58:10.331671 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81625) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.331956 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81625) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.336124 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81624) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 04:58:10.336405 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81624) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.336691 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81624) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.336962 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81624) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.337232 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81624) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.337528 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81624) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.337808 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81624) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.338081 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81624) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.338360 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81624) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.338633 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81624) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.338899 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81625) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 04:58:10.338986 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81624) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.339172 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81624) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.339265 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81625) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.339430 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81624) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.339522 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81625) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.339700 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81624) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.339795 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81625) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.339999 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81625) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.340284 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81625) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.340559 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81625) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.340838 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81625) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.341128 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81625) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.341408 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81625) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.341680 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81625) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.341943 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81625) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.342209 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81625) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.342479 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81625) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:58:10.374429 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x555ec70ab9c0 pid: 81624 (default app) Aug 30 04:58:10.377785 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd177e0c-289c-4edd-ae6d-85741aa0782a None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:10.377872 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x555ec70ab9c0 pid: 81625 (default app) Aug 30 04:58:10.378070 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd177e0c-289c-4edd-ae6d-85741aa0782a None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:10.378346 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd177e0c-289c-4edd-ae6d-85741aa0782a None None] PATH_INFO: `/v3` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:10.382473 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1/1] 10.222.0.77 () {50 vars in 791 bytes} [Tue Aug 30 04:58:10 2022] GET /identity/v3/ => generated 255 bytes in 7 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:58:16.854704 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc463bc4-a3bd-49fa-943e-2346a79210f7 None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:16.855241 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc463bc4-a3bd-49fa-943e-2346a79210f7 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:16.855346 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc463bc4-a3bd-49fa-943e-2346a79210f7 None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:16.859785 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1/2] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:58:16 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 04:58:16.865075 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ee27a84-aab9-46a7-b463-7d1e41df6bef None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:16.865633 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ee27a84-aab9-46a7-b463-7d1e41df6bef None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:16.865633 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ee27a84-aab9-46a7-b463-7d1e41df6bef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:16.939106 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: INFO dbcounter [None req-3ee27a84-aab9-46a7-b463-7d1e41df6bef None None] Registered counter for database keystone Aug 30 04:58:16.958243 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG oslo_db.sqlalchemy.engines [None req-3ee27a84-aab9-46a7-b463-7d1e41df6bef None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81624) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Aug 30 04:58:16.964564 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writer thread running {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Aug 30 04:58:17.143923 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG passlib.handlers.bcrypt [None req-3ee27a84-aab9-46a7-b463-7d1e41df6bef None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=81624) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Aug 30 04:58:17.144239 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG passlib.handlers.bcrypt [None req-3ee27a84-aab9-46a7-b463-7d1e41df6bef None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81624) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Aug 30 04:58:17.202120 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-3ee27a84-aab9-46a7-b463-7d1e41df6bef None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:58:17.203742 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-3ee27a84-aab9-46a7-b463-7d1e41df6bef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:58:17.329406 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 2/3] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:58:16 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 465 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Aug 30 04:58:17.866534 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc463bc4-a3bd-49fa-943e-2346a79210f7 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:17.936267 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: INFO dbcounter [None req-dc463bc4-a3bd-49fa-943e-2346a79210f7 None None] Registered counter for database keystone Aug 30 04:58:17.958110 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG oslo_db.sqlalchemy.engines [None req-dc463bc4-a3bd-49fa-943e-2346a79210f7 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81625) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Aug 30 04:58:17.964197 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writer thread running {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Aug 30 04:58:18.083532 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc463bc4-a3bd-49fa-943e-2346a79210f7 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:18.087795 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e176b96-1bc3-45cf-b093-34689868088e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:18.088433 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e176b96-1bc3-45cf-b093-34689868088e admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:18.088713 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e176b96-1bc3-45cf-b093-34689868088e admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:18.088986 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e176b96-1bc3-45cf-b093-34689868088e admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:18.097051 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e176b96-1bc3-45cf-b093-34689868088e admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:18.148337 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: WARNING keystone.common.rbac_enforcer.enforcer [None req-5e176b96-1bc3-45cf-b093-34689868088e admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Aug 30 04:58:18.148628 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e176b96-1bc3-45cf-b093-34689868088e admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:18.157955 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 2/4] 10.222.0.77 () {58 vars in 1324 bytes} [Tue Aug 30 04:58:17 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 292 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Aug 30 04:58:18.163951 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ee27a84-aab9-46a7-b463-7d1e41df6bef None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:18.178191 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ee27a84-aab9-46a7-b463-7d1e41df6bef None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:18.183056 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6826568f-b2b8-4b72-b9c8-4bbb2fd12c41 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:18.184163 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6826568f-b2b8-4b72-b9c8-4bbb2fd12c41 admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:18.184441 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6826568f-b2b8-4b72-b9c8-4bbb2fd12c41 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:18.184711 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6826568f-b2b8-4b72-b9c8-4bbb2fd12c41 admin admin] PATH_INFO: `/v3/projects/46478c740f59492585d5babfa714b5b5` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:18.186248 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6826568f-b2b8-4b72-b9c8-4bbb2fd12c41 admin admin] RBAC: Authorizing `identity:get_project(project_id=46478c740f59492585d5babfa714b5b5)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:18.238455 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: WARNING keystone.common.rbac_enforcer.enforcer [None req-6826568f-b2b8-4b72-b9c8-4bbb2fd12c41 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Aug 30 04:58:18.238792 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6826568f-b2b8-4b72-b9c8-4bbb2fd12c41 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:18.243352 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 3/5] 10.222.0.77 () {56 vars in 1207 bytes} [Tue Aug 30 04:58:18 2022] GET /identity/v3/projects/46478c740f59492585d5babfa714b5b5 => generated 344 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:58:20.106191 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d706de1-5575-409b-b92f-f54b50bc807c None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:20.106721 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d706de1-5575-409b-b92f-f54b50bc807c None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:20.106721 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d706de1-5575-409b-b92f-f54b50bc807c None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:20.110886 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 3/6] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:58:20 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 04:58:20.116202 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c1af2d5-6c23-42f0-b124-0b7823f5a06b None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:20.116568 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c1af2d5-6c23-42f0-b124-0b7823f5a06b None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:20.116754 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c1af2d5-6c23-42f0-b124-0b7823f5a06b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:20.136924 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-0c1af2d5-6c23-42f0-b124-0b7823f5a06b None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:58:20.138422 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-0c1af2d5-6c23-42f0-b124-0b7823f5a06b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:58:20.146100 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 4/7] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:58:20 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Aug 30 04:58:20.410701 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d706de1-5575-409b-b92f-f54b50bc807c None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:20.423330 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d706de1-5575-409b-b92f-f54b50bc807c None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:20.427545 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-555b06be-8e62-438c-a43d-a6f947226ba0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:20.428138 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-555b06be-8e62-438c-a43d-a6f947226ba0 admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:20.428423 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-555b06be-8e62-438c-a43d-a6f947226ba0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:20.428697 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-555b06be-8e62-438c-a43d-a6f947226ba0 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:20.436305 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-555b06be-8e62-438c-a43d-a6f947226ba0 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:20.438072 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-555b06be-8e62-438c-a43d-a6f947226ba0 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:20.446842 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 4/8] 10.222.0.77 () {58 vars in 1324 bytes} [Tue Aug 30 04:58:20 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 30 04:58:20.452063 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c1af2d5-6c23-42f0-b124-0b7823f5a06b None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:20.461146 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c1af2d5-6c23-42f0-b124-0b7823f5a06b None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:20.466015 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1195257b-737c-4baa-bcc9-4f23e02a76a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:20.466630 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1195257b-737c-4baa-bcc9-4f23e02a76a9 admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:20.466939 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1195257b-737c-4baa-bcc9-4f23e02a76a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:20.467211 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1195257b-737c-4baa-bcc9-4f23e02a76a9 admin admin] PATH_INFO: `/v3/users/fdab92bac3f64739a0fcb5f729d40f85` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:20.468505 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1195257b-737c-4baa-bcc9-4f23e02a76a9 admin admin] RBAC: Authorizing `identity:get_user(user_id=fdab92bac3f64739a0fcb5f729d40f85)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:20.470321 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1195257b-737c-4baa-bcc9-4f23e02a76a9 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:20.471882 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 5/9] 10.222.0.77 () {56 vars in 1198 bytes} [Tue Aug 30 04:58:20 2022] GET /identity/v3/users/fdab92bac3f64739a0fcb5f729d40f85 => generated 247 bytes in 20 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:58:22.407131 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec84ec6e-0b2b-4fd4-a21c-eb171eed7bea None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:22.407646 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec84ec6e-0b2b-4fd4-a21c-eb171eed7bea None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:22.407646 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec84ec6e-0b2b-4fd4-a21c-eb171eed7bea None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:22.408975 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 5/10] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:58:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:58:22.414093 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-712a2855-46f3-414e-a0b4-cd7811399b86 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:22.414492 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-712a2855-46f3-414e-a0b4-cd7811399b86 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:22.414756 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-712a2855-46f3-414e-a0b4-cd7811399b86 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:22.433074 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-712a2855-46f3-414e-a0b4-cd7811399b86 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:58:22.434251 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-712a2855-46f3-414e-a0b4-cd7811399b86 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:58:22.448062 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cfca864-3c28-4546-aee6-89ff71ddcb0a None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:22.448348 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cfca864-3c28-4546-aee6-89ff71ddcb0a None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:22.448627 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cfca864-3c28-4546-aee6-89ff71ddcb0a None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:22.449949 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 6/11] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:58:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:58:22.455146 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0007c2c7-de27-45aa-82eb-65ee77f838b1 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:22.455429 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0007c2c7-de27-45aa-82eb-65ee77f838b1 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:22.455700 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0007c2c7-de27-45aa-82eb-65ee77f838b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:22.497581 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG passlib.handlers.bcrypt [None req-0007c2c7-de27-45aa-82eb-65ee77f838b1 None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=81625) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Aug 30 04:58:22.497974 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG passlib.handlers.bcrypt [None req-0007c2c7-de27-45aa-82eb-65ee77f838b1 None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81625) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Aug 30 04:58:22.515210 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 6/12] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:58:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:58:22.546671 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-0007c2c7-de27-45aa-82eb-65ee77f838b1 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:58:22.548303 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-0007c2c7-de27-45aa-82eb-65ee77f838b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:58:22.646686 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 7/13] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:58:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 30 04:58:22.782173 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-712a2855-46f3-414e-a0b4-cd7811399b86 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:22.805747 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:22.888028 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-712a2855-46f3-414e-a0b4-cd7811399b86 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:22.908619 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0007c2c7-de27-45aa-82eb-65ee77f838b1 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:22.921218 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5068148-4b6b-4579-b053-7d8df46d31de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:22.921858 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5068148-4b6b-4579-b053-7d8df46d31de None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:22.922139 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5068148-4b6b-4579-b053-7d8df46d31de None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:22.922415 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5068148-4b6b-4579-b053-7d8df46d31de None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:22.933862 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5068148-4b6b-4579-b053-7d8df46d31de None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:58:22.942677 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-e5068148-4b6b-4579-b053-7d8df46d31de None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 04:58:22.942677 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:58:22.942677 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:58:22.942677 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:58:22.942677 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:58:22.942677 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:58:22.942677 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:58:22.942677 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:58:22.942677 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:58:22.942677 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:58:22.942677 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:58:22.942677 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:58:22.942677 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:58:22.942677 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:58:22.942677 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:58:22.942677 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:58:22.943808 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:58:22.943808 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:58:22.943808 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:58:22.943808 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:58:22.943808 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:58:22.943808 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:58:22.943808 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:58:22.943808 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:58:22.943808 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:58:22.943808 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:58:22.943808 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:58:22.943808 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 04:58:22.943808 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:58:22.943808 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:58:22.943808 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:58:22.943808 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:58:22.944757 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:58:22.944757 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:58:22.944757 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:58:22.944757 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:58:22.944757 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:58:22.944757 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:58:22.944757 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:58:22.944757 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:58:22.944757 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:58:22.944757 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:58:22.944757 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 04:58:22.944757 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 04:58:22.944757 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 7/14] 10.222.0.77 () {56 vars in 1096 bytes} [Tue Aug 30 04:58:22 2022] GET /identity/v3/roles/admin => generated 83 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:58:22.949404 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5068148-4b6b-4579-b053-7d8df46d31de None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:23.008455 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0007c2c7-de27-45aa-82eb-65ee77f838b1 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:23.010675 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:23.029958 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:23.047733 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed9b9881-d8a5-47bc-b1bd-2ce40a3ea31c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:23.048598 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed9b9881-d8a5-47bc-b1bd-2ce40a3ea31c None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:23.048693 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5068148-4b6b-4579-b053-7d8df46d31de None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:23.048878 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed9b9881-d8a5-47bc-b1bd-2ce40a3ea31c None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:23.049147 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed9b9881-d8a5-47bc-b1bd-2ce40a3ea31c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:23.081908 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3148da76-4754-40c3-a3fd-87c6dfeb7013 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:23.082499 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3148da76-4754-40c3-a3fd-87c6dfeb7013 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:23.082773 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3148da76-4754-40c3-a3fd-87c6dfeb7013 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:23.083073 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3148da76-4754-40c3-a3fd-87c6dfeb7013 None admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:23.083725 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3148da76-4754-40c3-a3fd-87c6dfeb7013 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:58:23.084019 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3148da76-4754-40c3-a3fd-87c6dfeb7013 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:23.085393 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3148da76-4754-40c3-a3fd-87c6dfeb7013 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:23.097013 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 8/15] 10.222.0.77 () {56 vars in 1099 bytes} [Tue Aug 30 04:58:22 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:58:23.102481 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3148da76-4754-40c3-a3fd-87c6dfeb7013 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:23.116199 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed9b9881-d8a5-47bc-b1bd-2ce40a3ea31c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:23.117718 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed9b9881-d8a5-47bc-b1bd-2ce40a3ea31c None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:23.200643 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3148da76-4754-40c3-a3fd-87c6dfeb7013 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:23.214717 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 8/16] 10.222.0.77 () {58 vars in 1282 bytes} [Tue Aug 30 04:58:22 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Aug 30 04:58:23.220199 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed9b9881-d8a5-47bc-b1bd-2ce40a3ea31c None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:23.222888 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:23.236282 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00a16007-dc9f-4d16-a06c-d06b1d37cf3e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:23.236905 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00a16007-dc9f-4d16-a06c-d06b1d37cf3e None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:23.237186 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00a16007-dc9f-4d16-a06c-d06b1d37cf3e None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:23.237459 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00a16007-dc9f-4d16-a06c-d06b1d37cf3e None admin] PATH_INFO: `/v3/users/fdab92bac3f64739a0fcb5f729d40f85` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:23.238589 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00a16007-dc9f-4d16-a06c-d06b1d37cf3e None admin] RBAC: Authorizing `identity:get_user(user_id=fdab92bac3f64739a0fcb5f729d40f85)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:23.240081 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00a16007-dc9f-4d16-a06c-d06b1d37cf3e None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:23.241653 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 9/17] 10.222.0.77 () {56 vars in 1177 bytes} [Tue Aug 30 04:58:23 2022] GET /identity/v3/users/fdab92bac3f64739a0fcb5f729d40f85 => generated 247 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:58:23.247089 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00a16007-dc9f-4d16-a06c-d06b1d37cf3e None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:23.264366 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:23.318982 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed9b9881-d8a5-47bc-b1bd-2ce40a3ea31c None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:23.348589 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00a16007-dc9f-4d16-a06c-d06b1d37cf3e None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:23.352447 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5218f016-eb65-40cd-b4a6-f6d22f871b80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:23.353817 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5218f016-eb65-40cd-b4a6-f6d22f871b80 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:23.354358 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5218f016-eb65-40cd-b4a6-f6d22f871b80 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:23.354624 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5218f016-eb65-40cd-b4a6-f6d22f871b80 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:23.373274 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5218f016-eb65-40cd-b4a6-f6d22f871b80 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:23.374733 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5218f016-eb65-40cd-b4a6-f6d22f871b80 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:23.383882 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c514b16-cffe-4d9a-ae70-318027953238 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:23.384515 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c514b16-cffe-4d9a-ae70-318027953238 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:23.384789 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c514b16-cffe-4d9a-ae70-318027953238 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:23.385056 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c514b16-cffe-4d9a-ae70-318027953238 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:23.386154 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c514b16-cffe-4d9a-ae70-318027953238 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:23.387609 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c514b16-cffe-4d9a-ae70-318027953238 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:23.389123 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 10/18] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 04:58:23 2022] GET /identity/v3/domains/default => generated 199 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:58:23.393036 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-5218f016-eb65-40cd-b4a6-f6d22f871b80 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:58:23.393036 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:58:23.393036 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:58:23.393036 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:58:23.393036 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:58:23.393036 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:58:23.393036 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:58:23.393036 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:58:23.393036 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:58:23.393036 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:58:23.393036 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:58:23.393036 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:58:23.393036 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:58:23.393036 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:58:23.393036 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:58:23.393036 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:58:23.393991 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:58:23.393991 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:58:23.393991 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:58:23.393991 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:58:23.393991 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:58:23.393991 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:58:23.393991 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:58:23.393991 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 04:58:23.393991 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:58:23.393991 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:58:23.393991 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:58:23.393991 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:58:23.393991 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:58:23.393991 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:58:23.393991 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:58:23.393991 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:58:23.393991 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:58:23.394789 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:58:23.394789 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:58:23.394789 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 04:58:23.394789 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 9/19] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 04:58:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 04:58:23.395397 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c514b16-cffe-4d9a-ae70-318027953238 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:23.400410 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5218f016-eb65-40cd-b4a6-f6d22f871b80 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:23.455933 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:23.477551 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:23.497963 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c514b16-cffe-4d9a-ae70-318027953238 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:23.503216 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5218f016-eb65-40cd-b4a6-f6d22f871b80 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:23.531838 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1e76a1b-7a01-47f8-9730-a369b7e1d7b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:23.532441 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1e76a1b-7a01-47f8-9730-a369b7e1d7b3 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:23.532720 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1e76a1b-7a01-47f8-9730-a369b7e1d7b3 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:23.532997 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1e76a1b-7a01-47f8-9730-a369b7e1d7b3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:23.533713 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1e76a1b-7a01-47f8-9730-a369b7e1d7b3 None admin] RBAC: Adding query filter params (role.id=9d9d5d2c270146488249b7ed943280db, scope.domain.id=default, user.id=fdab92bac3f64739a0fcb5f729d40f85) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:58:23.534015 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1e76a1b-7a01-47f8-9730-a369b7e1d7b3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:23.535499 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1e76a1b-7a01-47f8-9730-a369b7e1d7b3 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:23.537586 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6600dce5-f877-40b8-8b71-0e248d6a891a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:23.538184 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6600dce5-f877-40b8-8b71-0e248d6a891a None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:23.538462 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6600dce5-f877-40b8-8b71-0e248d6a891a None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:23.538731 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6600dce5-f877-40b8-8b71-0e248d6a891a None admin] PATH_INFO: `/v3/domains` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:23.539405 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6600dce5-f877-40b8-8b71-0e248d6a891a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:58:23.539703 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6600dce5-f877-40b8-8b71-0e248d6a891a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:23.541066 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6600dce5-f877-40b8-8b71-0e248d6a891a None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:23.542065 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 11/20] 10.222.0.77 () {56 vars in 1322 bytes} [Tue Aug 30 04:58:23 2022] GET /identity/v3/role_assignments?user.id=fdab92bac3f64739a0fcb5f729d40f85&scope.domain.id=default&role.id=9d9d5d2c270146488249b7ed943280db => generated 233 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:58:23.559748 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 10/21] 10.222.0.77 () {56 vars in 1109 bytes} [Tue Aug 30 04:58:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:58:25.669597 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc950c4a-08f7-42b2-8180-90f9468c2a0a None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:25.670639 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc950c4a-08f7-42b2-8180-90f9468c2a0a None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:25.671142 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc950c4a-08f7-42b2-8180-90f9468c2a0a None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:25.672660 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 12/22] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:58:25 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:58:25.678035 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1457461-2495-4f65-8234-2eae431db8f9 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:25.678890 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1457461-2495-4f65-8234-2eae431db8f9 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:25.679451 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1457461-2495-4f65-8234-2eae431db8f9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:25.699321 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92370c0c-2abf-46f3-943b-8732c08eb465 None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:25.699946 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92370c0c-2abf-46f3-943b-8732c08eb465 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:25.700416 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92370c0c-2abf-46f3-943b-8732c08eb465 None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:25.701981 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 13/23] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:58:25 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:58:25.707022 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ec9432e-490e-4823-b549-60d830129e92 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:25.707563 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ec9432e-490e-4823-b549-60d830129e92 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:25.708046 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ec9432e-490e-4823-b549-60d830129e92 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:25.708326 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-e1457461-2495-4f65-8234-2eae431db8f9 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:58:25.709527 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-e1457461-2495-4f65-8234-2eae431db8f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:58:25.738645 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-1ec9432e-490e-4823-b549-60d830129e92 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:58:25.740188 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-1ec9432e-490e-4823-b549-60d830129e92 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:58:25.845050 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 11/24] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:58:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:58:25.873086 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 14/25] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:58:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 30 04:58:26.238300 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1457461-2495-4f65-8234-2eae431db8f9 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:26.293263 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:26.363308 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1457461-2495-4f65-8234-2eae431db8f9 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:26.394204 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ada24c5b-bfb6-42d2-95c2-03dd2ceb69a5 None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:26.394861 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ada24c5b-bfb6-42d2-95c2-03dd2ceb69a5 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:26.395411 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ada24c5b-bfb6-42d2-95c2-03dd2ceb69a5 None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:26.396940 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 15/26] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:58:26 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 04:58:26.408341 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-596c8c39-8075-459d-aa26-a47dea79502d None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:26.408878 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-596c8c39-8075-459d-aa26-a47dea79502d None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:26.409346 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-596c8c39-8075-459d-aa26-a47dea79502d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:26.415781 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fde9a90-53d5-4299-8837-ee46155fa992 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:26.416672 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fde9a90-53d5-4299-8837-ee46155fa992 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:26.417164 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fde9a90-53d5-4299-8837-ee46155fa992 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:26.417631 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fde9a90-53d5-4299-8837-ee46155fa992 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:26.435244 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-596c8c39-8075-459d-aa26-a47dea79502d None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:58:26.436741 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-596c8c39-8075-459d-aa26-a47dea79502d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:58:26.447555 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fde9a90-53d5-4299-8837-ee46155fa992 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:26.449307 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fde9a90-53d5-4299-8837-ee46155fa992 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:26.482175 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-0fde9a90-53d5-4299-8837-ee46155fa992 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:58:26.482175 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:58:26.482175 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:58:26.482175 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:58:26.482175 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:58:26.482175 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:58:26.482175 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:58:26.482175 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:58:26.482175 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:58:26.482175 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:58:26.482175 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:58:26.482175 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:58:26.482175 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:58:26.482175 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:58:26.482175 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:58:26.482175 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:58:26.483075 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:58:26.483075 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:58:26.483075 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:58:26.483075 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:58:26.483075 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:58:26.483075 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:58:26.483075 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:58:26.483075 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 04:58:26.483075 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:58:26.483075 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:58:26.483075 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:58:26.483075 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:58:26.483075 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:58:26.483075 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:58:26.483075 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:58:26.483075 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:58:26.483075 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:58:26.483860 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:58:26.483860 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:58:26.483860 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 04:58:26.484450 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 12/27] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 04:58:26 2022] GET /identity/v3/domains/Default => generated 87 bytes in 246 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:58:26.484956 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fde9a90-53d5-4299-8837-ee46155fa992 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:26.555233 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 16/28] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:58:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 30 04:58:26.556623 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-596c8c39-8075-459d-aa26-a47dea79502d None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:26.560772 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:26.657776 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-596c8c39-8075-459d-aa26-a47dea79502d None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:26.693538 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d23ad1ea-6658-4406-96a5-e827e7f1d80c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:26.694811 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d23ad1ea-6658-4406-96a5-e827e7f1d80c None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:26.695305 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d23ad1ea-6658-4406-96a5-e827e7f1d80c None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:26.695721 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d23ad1ea-6658-4406-96a5-e827e7f1d80c None admin] PATH_INFO: `/v3/domains` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:26.696534 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d23ad1ea-6658-4406-96a5-e827e7f1d80c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:58:26.697076 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d23ad1ea-6658-4406-96a5-e827e7f1d80c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:26.698594 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d23ad1ea-6658-4406-96a5-e827e7f1d80c None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:26.715955 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 17/29] 10.222.0.77 () {56 vars in 1109 bytes} [Tue Aug 30 04:58:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:58:26.717453 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2dd9139-3bb8-4c4c-86a6-0f2d3dd212e2 None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:26.717884 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2dd9139-3bb8-4c4c-86a6-0f2d3dd212e2 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:26.718307 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2dd9139-3bb8-4c4c-86a6-0f2d3dd212e2 None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:26.719987 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 18/30] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:58:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:58:26.732375 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2dd9139-3bb8-4c4c-86a6-0f2d3dd212e2 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:26.738153 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:26.743027 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fde9a90-53d5-4299-8837-ee46155fa992 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:26.786252 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c8ce883-afe5-4362-882d-6cff8e00574d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:26.786901 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c8ce883-afe5-4362-882d-6cff8e00574d None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:26.787706 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c8ce883-afe5-4362-882d-6cff8e00574d None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:26.789300 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c8ce883-afe5-4362-882d-6cff8e00574d None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:26.822455 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c8ce883-afe5-4362-882d-6cff8e00574d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:58:26.828556 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-7c8ce883-afe5-4362-882d-6cff8e00574d None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 04:58:26.828556 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:58:26.828556 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:58:26.828556 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:58:26.828556 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:58:26.828556 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:58:26.828556 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:58:26.828556 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:58:26.828556 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:58:26.828556 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:58:26.828556 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:58:26.828556 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:58:26.828556 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:58:26.828556 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:58:26.828556 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:58:26.828556 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:58:26.829376 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:58:26.829376 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:58:26.829376 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:58:26.829376 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:58:26.829376 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:58:26.829376 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:58:26.829376 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:58:26.829376 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:58:26.829376 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:58:26.829376 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:58:26.829376 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:58:26.829376 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 04:58:26.829376 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:58:26.829376 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:58:26.829376 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:58:26.829376 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:58:26.830118 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:58:26.830118 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:58:26.830118 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:58:26.830118 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:58:26.830118 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:58:26.830118 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:58:26.830118 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:58:26.830118 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:58:26.830118 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:58:26.830118 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:58:26.830118 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 04:58:26.830118 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 04:58:26.830118 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 13/31] 10.222.0.77 () {56 vars in 1096 bytes} [Tue Aug 30 04:58:26 2022] GET /identity/v3/roles/admin => generated 83 bytes in 345 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:58:26.831474 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd6ab9e0-0f22-446c-a406-5daff7c35e18 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:26.831593 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:26.839316 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd6ab9e0-0f22-446c-a406-5daff7c35e18 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:26.839633 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd6ab9e0-0f22-446c-a406-5daff7c35e18 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:26.868152 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-fd6ab9e0-0f22-446c-a406-5daff7c35e18 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:58:26.869667 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2dd9139-3bb8-4c4c-86a6-0f2d3dd212e2 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:26.870509 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-fd6ab9e0-0f22-446c-a406-5daff7c35e18 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:58:26.905541 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a46642ba-8d75-40b5-a9bb-55bb05e5a68f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:26.906233 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a46642ba-8d75-40b5-a9bb-55bb05e5a68f None admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:26.906677 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a46642ba-8d75-40b5-a9bb-55bb05e5a68f None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:26.907117 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a46642ba-8d75-40b5-a9bb-55bb05e5a68f None admin] PATH_INFO: `/v3/projects` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:26.907905 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a46642ba-8d75-40b5-a9bb-55bb05e5a68f None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:26.909546 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a46642ba-8d75-40b5-a9bb-55bb05e5a68f None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:26.942265 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 14/32] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:58:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:58:26.943756 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd6ab9e0-0f22-446c-a406-5daff7c35e18 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:27.111279 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:27.129954 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd6ab9e0-0f22-446c-a406-5daff7c35e18 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:27.166722 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 19/33] 10.222.0.77 () {60 vars in 1141 bytes} [Tue Aug 30 04:58:26 2022] POST /identity/v3/projects => generated 301 bytes in 435 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:58:27.168040 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a46642ba-8d75-40b5-a9bb-55bb05e5a68f None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:27.203409 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-465cf5a8-166f-4884-bb2e-2abdd9891173 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:27.205384 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-465cf5a8-166f-4884-bb2e-2abdd9891173 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:27.206618 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-465cf5a8-166f-4884-bb2e-2abdd9891173 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:27.207159 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-465cf5a8-166f-4884-bb2e-2abdd9891173 None admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:27.208335 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-465cf5a8-166f-4884-bb2e-2abdd9891173 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:58:27.209581 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-465cf5a8-166f-4884-bb2e-2abdd9891173 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:27.212606 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-465cf5a8-166f-4884-bb2e-2abdd9891173 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:27.238112 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 15/34] 10.222.0.77 () {56 vars in 1099 bytes} [Tue Aug 30 04:58:26 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 295 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:58:27.239381 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1cb8783-7888-408c-a065-4321c1b188d6 None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:27.239676 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1cb8783-7888-408c-a065-4321c1b188d6 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:27.239945 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1cb8783-7888-408c-a065-4321c1b188d6 None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:27.241399 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 16/35] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:58:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:58:27.246452 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1cb8783-7888-408c-a065-4321c1b188d6 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:27.319559 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a46642ba-8d75-40b5-a9bb-55bb05e5a68f None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:27.339449 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:27.366299 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1cb8783-7888-408c-a065-4321c1b188d6 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:27.366773 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67c5431c-be23-4ed0-aaab-e6d70ab6d518 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:27.367444 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c5431c-be23-4ed0-aaab-e6d70ab6d518 None admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:27.367885 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c5431c-be23-4ed0-aaab-e6d70ab6d518 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:27.368169 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c5431c-be23-4ed0-aaab-e6d70ab6d518 None admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:27.369721 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67c5431c-be23-4ed0-aaab-e6d70ab6d518 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:27.371641 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67c5431c-be23-4ed0-aaab-e6d70ab6d518 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:27.395220 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 20/36] 10.222.0.77 () {60 vars in 1131 bytes} [Tue Aug 30 04:58:27 2022] POST /identity/v3/roles => generated 219 bytes in 228 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:58:27.396152 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67c5431c-be23-4ed0-aaab-e6d70ab6d518 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:27.413271 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c70f00d-583a-4c8c-848a-b8f39b6ef9d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:27.414474 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c70f00d-583a-4c8c-848a-b8f39b6ef9d8 None admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:27.414881 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c70f00d-583a-4c8c-848a-b8f39b6ef9d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:27.415201 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c70f00d-583a-4c8c-848a-b8f39b6ef9d8 None admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:27.415942 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c70f00d-583a-4c8c-848a-b8f39b6ef9d8 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:27.417377 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c70f00d-583a-4c8c-848a-b8f39b6ef9d8 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:27.523571 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67c5431c-be23-4ed0-aaab-e6d70ab6d518 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:27.545652 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 17/37] 10.222.0.77 () {60 vars in 1131 bytes} [Tue Aug 30 04:58:27 2022] POST /identity/v3/roles => generated 225 bytes in 304 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:58:27.547125 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf2253c3-0318-4b4c-b7ce-2943def60777 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:27.547421 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf2253c3-0318-4b4c-b7ce-2943def60777 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:27.547700 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf2253c3-0318-4b4c-b7ce-2943def60777 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:27.568879 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-cf2253c3-0318-4b4c-b7ce-2943def60777 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:58:27.570499 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-cf2253c3-0318-4b4c-b7ce-2943def60777 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:58:27.572955 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e636bdcf-e5dd-4278-a320-3de2809e6b43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:27.573666 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e636bdcf-e5dd-4278-a320-3de2809e6b43 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:27.574530 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e636bdcf-e5dd-4278-a320-3de2809e6b43 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:27.574530 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e636bdcf-e5dd-4278-a320-3de2809e6b43 None admin] PATH_INFO: `/v3/users/fdab92bac3f64739a0fcb5f729d40f85` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:27.576540 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e636bdcf-e5dd-4278-a320-3de2809e6b43 None admin] RBAC: Authorizing `identity:get_user(user_id=fdab92bac3f64739a0fcb5f729d40f85)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:27.576929 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:27.579535 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e636bdcf-e5dd-4278-a320-3de2809e6b43 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:27.581337 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 21/38] 10.222.0.77 () {56 vars in 1177 bytes} [Tue Aug 30 04:58:27 2022] GET /identity/v3/users/fdab92bac3f64739a0fcb5f729d40f85 => generated 247 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:58:27.581510 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eedb23c-2b02-44a9-a9d6-273e43fecba7 None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:27.582240 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eedb23c-2b02-44a9-a9d6-273e43fecba7 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:27.582240 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eedb23c-2b02-44a9-a9d6-273e43fecba7 None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:27.583485 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 22/39] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:58:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:58:27.589440 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eedb23c-2b02-44a9-a9d6-273e43fecba7 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:27.655957 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:27.667223 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 18/40] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:58:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:58:27.676343 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0a39aca-5803-4445-af85-050ff3b53fb7 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:27.676648 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0a39aca-5803-4445-af85-050ff3b53fb7 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:27.676919 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0a39aca-5803-4445-af85-050ff3b53fb7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:27.706100 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-f0a39aca-5803-4445-af85-050ff3b53fb7 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:58:27.707807 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-f0a39aca-5803-4445-af85-050ff3b53fb7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:58:27.722420 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eedb23c-2b02-44a9-a9d6-273e43fecba7 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:27.767670 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef651d43-3ef6-4274-8b5f-15c8594e6bd4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:27.768655 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef651d43-3ef6-4274-8b5f-15c8594e6bd4 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:27.769056 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef651d43-3ef6-4274-8b5f-15c8594e6bd4 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:27.769337 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef651d43-3ef6-4274-8b5f-15c8594e6bd4 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:27.770529 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef651d43-3ef6-4274-8b5f-15c8594e6bd4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:27.772024 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef651d43-3ef6-4274-8b5f-15c8594e6bd4 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:27.773769 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 23/41] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 04:58:27 2022] GET /identity/v3/domains/default => generated 199 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:58:27.779986 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef651d43-3ef6-4274-8b5f-15c8594e6bd4 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:27.797518 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 19/42] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:58:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:58:27.942695 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0a39aca-5803-4445-af85-050ff3b53fb7 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:27.996602 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef651d43-3ef6-4274-8b5f-15c8594e6bd4 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:28.011270 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:28.022170 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:28.035009 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ed9bee5-2ebe-4432-bdf1-8b67abeb3159 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:28.035670 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed9bee5-2ebe-4432-bdf1-8b67abeb3159 None admin] REQUEST_METHOD: `PUT` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:28.035952 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed9bee5-2ebe-4432-bdf1-8b67abeb3159 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:28.036222 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed9bee5-2ebe-4432-bdf1-8b67abeb3159 None admin] PATH_INFO: `/v3/domains/default/users/fdab92bac3f64739a0fcb5f729d40f85/roles/9d9d5d2c270146488249b7ed943280db` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:28.038132 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ed9bee5-2ebe-4432-bdf1-8b67abeb3159 None admin] RBAC: Authorizing `identity:create_grant(domain_id=default, user_id=fdab92bac3f64739a0fcb5f729d40f85, role_id=9d9d5d2c270146488249b7ed943280db)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:28.039599 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ed9bee5-2ebe-4432-bdf1-8b67abeb3159 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:28.055448 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 24/43] 10.222.0.77 () {58 vars in 1361 bytes} [Tue Aug 30 04:58:27 2022] PUT /identity/v3/domains/default/users/fdab92bac3f64739a0fcb5f729d40f85/roles/9d9d5d2c270146488249b7ed943280db => generated 0 bytes in 276 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:58:28.058633 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0a39aca-5803-4445-af85-050ff3b53fb7 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:28.066734 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ed9bee5-2ebe-4432-bdf1-8b67abeb3159 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:28.103785 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa69cb5e-47e1-4b7a-b73c-c1694bf170af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:28.105181 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa69cb5e-47e1-4b7a-b73c-c1694bf170af None admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:28.105654 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa69cb5e-47e1-4b7a-b73c-c1694bf170af None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:28.106125 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa69cb5e-47e1-4b7a-b73c-c1694bf170af None admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:28.106870 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa69cb5e-47e1-4b7a-b73c-c1694bf170af None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:28.108800 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa69cb5e-47e1-4b7a-b73c-c1694bf170af None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:28.127603 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 20/44] 10.222.0.77 () {60 vars in 1131 bytes} [Tue Aug 30 04:58:27 2022] POST /identity/v3/roles => generated 223 bytes in 185 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 04:58:28.188693 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ed9bee5-2ebe-4432-bdf1-8b67abeb3159 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:28.230964 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86e996e3-3141-4d8e-982c-ef42cc8c1dc8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:28.231498 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86e996e3-3141-4d8e-982c-ef42cc8c1dc8 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:28.231777 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86e996e3-3141-4d8e-982c-ef42cc8c1dc8 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:28.232051 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86e996e3-3141-4d8e-982c-ef42cc8c1dc8 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:28.233208 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86e996e3-3141-4d8e-982c-ef42cc8c1dc8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:28.234710 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86e996e3-3141-4d8e-982c-ef42cc8c1dc8 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:28.236527 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 25/45] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 04:58:28 2022] GET /identity/v3/domains/default => generated 199 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:58:28.242294 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa69cb5e-47e1-4b7a-b73c-c1694bf170af None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:28.355082 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa69cb5e-47e1-4b7a-b73c-c1694bf170af None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:28.382830 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:28.389760 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-757eef1e-6b0f-4b0d-9056-ebe65e07ac2b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:28.390526 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-757eef1e-6b0f-4b0d-9056-ebe65e07ac2b None admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:28.391089 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-757eef1e-6b0f-4b0d-9056-ebe65e07ac2b None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:28.391372 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-757eef1e-6b0f-4b0d-9056-ebe65e07ac2b None admin] PATH_INFO: `/v3/projects` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:28.392332 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-757eef1e-6b0f-4b0d-9056-ebe65e07ac2b None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:28.393794 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-757eef1e-6b0f-4b0d-9056-ebe65e07ac2b None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:28.421607 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 21/46] 10.222.0.77 () {60 vars in 1141 bytes} [Tue Aug 30 04:58:28 2022] POST /identity/v3/projects => generated 312 bytes in 180 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:58:29.955675 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1080ca0d-c84c-4723-9409-11d8f674379b None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:29.956213 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1080ca0d-c84c-4723-9409-11d8f674379b None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:29.956213 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1080ca0d-c84c-4723-9409-11d8f674379b None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:29.957550 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 26/47] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:58:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:58:29.962857 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf6c43a-0fc5-443c-abcc-9918045afa5b None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:29.963595 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf6c43a-0fc5-443c-abcc-9918045afa5b None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:29.963881 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf6c43a-0fc5-443c-abcc-9918045afa5b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:29.981971 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-faf6c43a-0fc5-443c-abcc-9918045afa5b None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:58:29.983191 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-faf6c43a-0fc5-443c-abcc-9918045afa5b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:58:30.049378 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 22/48] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:58:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:58:30.307875 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5d03b56-9bd5-4a3e-af59-aeb56717edb1 None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:30.308163 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5d03b56-9bd5-4a3e-af59-aeb56717edb1 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:30.308438 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5d03b56-9bd5-4a3e-af59-aeb56717edb1 None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:30.309789 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 27/49] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:58:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 04:58:30.314809 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5d03b56-9bd5-4a3e-af59-aeb56717edb1 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:30.314948 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a44d9a-a8a4-4121-a4b1-b0b2451cd608 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:30.315183 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a44d9a-a8a4-4121-a4b1-b0b2451cd608 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:30.315461 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a44d9a-a8a4-4121-a4b1-b0b2451cd608 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:30.332921 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-95a44d9a-a8a4-4121-a4b1-b0b2451cd608 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:58:30.334110 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-95a44d9a-a8a4-4121-a4b1-b0b2451cd608 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:58:30.358569 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:30.403309 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 23/50] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:58:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:58:30.417877 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5d03b56-9bd5-4a3e-af59-aeb56717edb1 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:30.450762 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49c3434a-a7f3-4625-9635-c85006b98aa4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:30.451410 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c3434a-a7f3-4625-9635-c85006b98aa4 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:30.451690 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c3434a-a7f3-4625-9635-c85006b98aa4 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:30.451960 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c3434a-a7f3-4625-9635-c85006b98aa4 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:30.464118 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49c3434a-a7f3-4625-9635-c85006b98aa4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:58:30.469006 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-49c3434a-a7f3-4625-9635-c85006b98aa4 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 04:58:30.469006 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:58:30.469006 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:58:30.469006 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:58:30.469006 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:58:30.469006 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:58:30.469006 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:58:30.469006 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:58:30.469006 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:58:30.469006 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:58:30.469006 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:58:30.469006 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:58:30.469006 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:58:30.469006 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:58:30.469006 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:58:30.469006 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:58:30.469797 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:58:30.469797 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:58:30.469797 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:58:30.469797 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:58:30.469797 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:58:30.469797 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:58:30.469797 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:58:30.469797 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:58:30.469797 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:58:30.469797 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:58:30.469797 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:58:30.469797 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 04:58:30.469797 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:58:30.469797 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:58:30.469797 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:58:30.469797 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:58:30.470545 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:58:30.470545 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:58:30.470545 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:58:30.470545 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:58:30.470545 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:58:30.470545 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:58:30.470545 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:58:30.470545 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:58:30.470545 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:58:30.470545 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:58:30.470545 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 04:58:30.470545 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 04:58:30.470545 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 28/51] 10.222.0.77 () {56 vars in 1096 bytes} [Tue Aug 30 04:58:30 2022] GET /identity/v3/roles/admin => generated 83 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 04:58:30.475840 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95a44d9a-a8a4-4121-a4b1-b0b2451cd608 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:30.519912 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:30.575628 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95a44d9a-a8a4-4121-a4b1-b0b2451cd608 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:30.609004 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e749d155-6b7a-4037-9b69-a6c7a5da8f7e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:30.609604 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e749d155-6b7a-4037-9b69-a6c7a5da8f7e None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:30.609886 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e749d155-6b7a-4037-9b69-a6c7a5da8f7e None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:30.610158 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e749d155-6b7a-4037-9b69-a6c7a5da8f7e None admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:30.610810 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e749d155-6b7a-4037-9b69-a6c7a5da8f7e None admin] RBAC: Adding query filter params (name=admin) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:58:30.611137 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e749d155-6b7a-4037-9b69-a6c7a5da8f7e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:30.612501 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e749d155-6b7a-4037-9b69-a6c7a5da8f7e None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:30.620858 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 24/52] 10.222.0.77 () {56 vars in 1099 bytes} [Tue Aug 30 04:58:30 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:58:30.626997 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49c3434a-a7f3-4625-9635-c85006b98aa4 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:30.670690 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e749d155-6b7a-4037-9b69-a6c7a5da8f7e None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:30.726746 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:30.730785 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49c3434a-a7f3-4625-9635-c85006b98aa4 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:30.766734 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dafbe68-a55d-4251-83a5-6db292cc1603 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:30.767404 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dafbe68-a55d-4251-83a5-6db292cc1603 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:30.767685 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dafbe68-a55d-4251-83a5-6db292cc1603 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:30.767958 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dafbe68-a55d-4251-83a5-6db292cc1603 None admin] PATH_INFO: `/v3/users/fdab92bac3f64739a0fcb5f729d40f85` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:30.769110 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dafbe68-a55d-4251-83a5-6db292cc1603 None admin] RBAC: Authorizing `identity:get_user(user_id=fdab92bac3f64739a0fcb5f729d40f85)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:30.770590 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dafbe68-a55d-4251-83a5-6db292cc1603 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:30.772238 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 29/53] 10.222.0.77 () {56 vars in 1177 bytes} [Tue Aug 30 04:58:30 2022] GET /identity/v3/users/fdab92bac3f64739a0fcb5f729d40f85 => generated 247 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:58:30.778522 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dafbe68-a55d-4251-83a5-6db292cc1603 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:30.873687 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e749d155-6b7a-4037-9b69-a6c7a5da8f7e None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:30.875902 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:30.877271 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dafbe68-a55d-4251-83a5-6db292cc1603 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:30.909135 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d2328b2-94a7-4543-a686-371c76c8b26a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:30.909759 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d2328b2-94a7-4543-a686-371c76c8b26a None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:30.910038 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d2328b2-94a7-4543-a686-371c76c8b26a None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:30.910305 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d2328b2-94a7-4543-a686-371c76c8b26a None admin] PATH_INFO: `/v3/domains/default` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:30.911421 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d2328b2-94a7-4543-a686-371c76c8b26a None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:30.912854 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d2328b2-94a7-4543-a686-371c76c8b26a None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:30.913271 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a70df38-fded-4216-a92a-e780c143a40c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:30.913898 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a70df38-fded-4216-a92a-e780c143a40c None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:30.914188 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a70df38-fded-4216-a92a-e780c143a40c None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:30.914397 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 25/54] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 04:58:30 2022] GET /identity/v3/domains/default => generated 199 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:58:30.914479 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a70df38-fded-4216-a92a-e780c143a40c None admin] PATH_INFO: `/v3/domains/default` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:30.916151 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a70df38-fded-4216-a92a-e780c143a40c None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:30.917594 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a70df38-fded-4216-a92a-e780c143a40c None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:30.919203 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 30/55] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 04:58:30 2022] GET /identity/v3/domains/default => generated 199 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:58:30.920354 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d2328b2-94a7-4543-a686-371c76c8b26a None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:30.924566 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a70df38-fded-4216-a92a-e780c143a40c None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:30.957562 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:31.017332 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d2328b2-94a7-4543-a686-371c76c8b26a None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:31.022653 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a70df38-fded-4216-a92a-e780c143a40c None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:31.049786 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29df6379-0bdd-421e-8a57-cd2016625879 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:31.050378 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29df6379-0bdd-421e-8a57-cd2016625879 None admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:31.050659 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29df6379-0bdd-421e-8a57-cd2016625879 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:31.050968 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29df6379-0bdd-421e-8a57-cd2016625879 None admin] PATH_INFO: `/v3/projects` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:31.051611 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29df6379-0bdd-421e-8a57-cd2016625879 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:31.053053 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29df6379-0bdd-421e-8a57-cd2016625879 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:31.056246 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-451d1a28-f732-4b2e-9b8f-62a665d46bfc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:31.056852 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-451d1a28-f732-4b2e-9b8f-62a665d46bfc None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:31.057137 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-451d1a28-f732-4b2e-9b8f-62a665d46bfc None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:31.057574 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-451d1a28-f732-4b2e-9b8f-62a665d46bfc None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:31.058312 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-451d1a28-f732-4b2e-9b8f-62a665d46bfc None admin] RBAC: Adding query filter params (role.id=9d9d5d2c270146488249b7ed943280db, scope.domain.id=default, user.id=fdab92bac3f64739a0fcb5f729d40f85) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:58:31.058624 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-451d1a28-f732-4b2e-9b8f-62a665d46bfc None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:31.060098 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-451d1a28-f732-4b2e-9b8f-62a665d46bfc None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:31.066157 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 31/56] 10.222.0.77 () {56 vars in 1322 bytes} [Tue Aug 30 04:58:30 2022] GET /identity/v3/role_assignments?user.id=fdab92bac3f64739a0fcb5f729d40f85&scope.domain.id=default&role.id=9d9d5d2c270146488249b7ed943280db => generated 530 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:58:31.079415 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 26/57] 10.222.0.77 () {60 vars in 1140 bytes} [Tue Aug 30 04:58:30 2022] POST /identity/v3/projects => generated 298 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:58:32.972120 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5501d255-db5e-4013-bfe5-6df6ac328286 None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:32.972625 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5501d255-db5e-4013-bfe5-6df6ac328286 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:32.972733 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5501d255-db5e-4013-bfe5-6df6ac328286 None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:32.974025 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 32/58] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:58:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:58:32.979905 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c6042de-9119-45d1-96a3-08e3ba35e10f None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:32.980298 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c6042de-9119-45d1-96a3-08e3ba35e10f None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:32.980456 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c6042de-9119-45d1-96a3-08e3ba35e10f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:32.998021 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-6c6042de-9119-45d1-96a3-08e3ba35e10f None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:58:32.999258 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-6c6042de-9119-45d1-96a3-08e3ba35e10f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:58:33.016029 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:33.065081 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 27/59] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:58:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:58:33.331889 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5501d255-db5e-4013-bfe5-6df6ac328286 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:33.427053 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:33.430912 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5501d255-db5e-4013-bfe5-6df6ac328286 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:33.463488 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e621d21-89a1-49e3-9833-b3af26bb66d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:33.464112 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e621d21-89a1-49e3-9833-b3af26bb66d8 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:33.464395 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e621d21-89a1-49e3-9833-b3af26bb66d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:33.464680 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e621d21-89a1-49e3-9833-b3af26bb66d8 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:33.466008 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e621d21-89a1-49e3-9833-b3af26bb66d8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:33.467245 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e621d21-89a1-49e3-9833-b3af26bb66d8 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:33.468796 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 33/60] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 04:58:33 2022] GET /identity/v3/domains/default => generated 199 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:58:33.474158 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c6042de-9119-45d1-96a3-08e3ba35e10f None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:33.568482 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c6042de-9119-45d1-96a3-08e3ba35e10f None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:33.601163 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df3a32fc-d4f1-4e70-9f89-6a5a15210177 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:33.601695 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df3a32fc-d4f1-4e70-9f89-6a5a15210177 None admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:33.601970 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df3a32fc-d4f1-4e70-9f89-6a5a15210177 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:33.602237 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df3a32fc-d4f1-4e70-9f89-6a5a15210177 None admin] PATH_INFO: `/v3/users` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:33.602923 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df3a32fc-d4f1-4e70-9f89-6a5a15210177 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:33.604613 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df3a32fc-d4f1-4e70-9f89-6a5a15210177 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:33.629498 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 28/61] 10.222.0.77 () {60 vars in 1132 bytes} [Tue Aug 30 04:58:33 2022] POST /identity/v3/users => generated 275 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:58:36.565169 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23db9924-b31e-4474-a5d8-ddf8075699fb None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:36.565671 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23db9924-b31e-4474-a5d8-ddf8075699fb None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:36.565743 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23db9924-b31e-4474-a5d8-ddf8075699fb None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:36.567192 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 34/62] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:58:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:58:36.572833 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec35e63-d0c5-4b2a-8490-abeda1c247e6 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:36.573252 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec35e63-d0c5-4b2a-8490-abeda1c247e6 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:36.573442 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec35e63-d0c5-4b2a-8490-abeda1c247e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:36.583466 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43afa257-91b4-4a41-af0d-a20b4c0c8512 None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:36.584017 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43afa257-91b4-4a41-af0d-a20b4c0c8512 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:36.584493 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43afa257-91b4-4a41-af0d-a20b4c0c8512 None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:36.585998 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 35/63] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:58:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 04:58:36.591964 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48dfabc1-249c-433b-8ea4-00f718584a98 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:36.592524 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48dfabc1-249c-433b-8ea4-00f718584a98 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:36.592999 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48dfabc1-249c-433b-8ea4-00f718584a98 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:36.601131 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-8ec35e63-d0c5-4b2a-8490-abeda1c247e6 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:58:36.602695 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-8ec35e63-d0c5-4b2a-8490-abeda1c247e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:58:36.606304 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:36.628165 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-48dfabc1-249c-433b-8ea4-00f718584a98 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:58:36.629709 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-48dfabc1-249c-433b-8ea4-00f718584a98 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:58:36.732740 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 29/64] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:58:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 30 04:58:36.761997 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 36/65] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:58:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 30 04:58:36.785428 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c0ae279-13dd-469f-b050-7588f3a2cf08 None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:36.786045 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c0ae279-13dd-469f-b050-7588f3a2cf08 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:36.786517 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c0ae279-13dd-469f-b050-7588f3a2cf08 None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:36.788105 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 30/66] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:58:36 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:58:36.793541 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1748e2e1-c772-470e-b44a-fbd942430791 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:36.793829 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1748e2e1-c772-470e-b44a-fbd942430791 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:36.794101 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1748e2e1-c772-470e-b44a-fbd942430791 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:36.828732 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-1748e2e1-c772-470e-b44a-fbd942430791 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:58:36.829912 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-1748e2e1-c772-470e-b44a-fbd942430791 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:58:36.932708 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 37/67] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:58:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:58:37.105788 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c0ae279-13dd-469f-b050-7588f3a2cf08 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:37.128429 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1748e2e1-c772-470e-b44a-fbd942430791 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:37.177674 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:37.271711 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c0ae279-13dd-469f-b050-7588f3a2cf08 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:37.309505 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1748e2e1-c772-470e-b44a-fbd942430791 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:37.323361 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e12f7ddf-dc73-483e-95e0-c64c99f5dcbf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:37.323995 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e12f7ddf-dc73-483e-95e0-c64c99f5dcbf None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:37.324283 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e12f7ddf-dc73-483e-95e0-c64c99f5dcbf None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:37.324558 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e12f7ddf-dc73-483e-95e0-c64c99f5dcbf None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:37.332222 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:37.335667 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e12f7ddf-dc73-483e-95e0-c64c99f5dcbf None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:58:37.340792 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-e12f7ddf-dc73-483e-95e0-c64c99f5dcbf None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 04:58:37.340792 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:58:37.340792 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:58:37.340792 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:58:37.340792 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:58:37.340792 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:58:37.340792 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:58:37.340792 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:58:37.340792 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:58:37.340792 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:58:37.340792 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:58:37.340792 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:58:37.340792 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:58:37.340792 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:58:37.340792 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:58:37.340792 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:58:37.341627 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:58:37.341627 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:58:37.341627 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:58:37.341627 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:58:37.341627 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:58:37.341627 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:58:37.341627 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:58:37.341627 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:58:37.341627 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:58:37.341627 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:58:37.341627 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:58:37.341627 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 04:58:37.341627 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:58:37.341627 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:58:37.341627 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:58:37.341627 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:58:37.342752 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:58:37.342752 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:58:37.342752 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:58:37.342752 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:58:37.342752 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:58:37.342752 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:58:37.342752 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:58:37.342752 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:58:37.342752 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:58:37.342752 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:58:37.342752 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 04:58:37.342752 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 04:58:37.342752 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 31/68] 10.222.0.77 () {56 vars in 1096 bytes} [Tue Aug 30 04:58:37 2022] GET /identity/v3/roles/admin => generated 83 bytes in 237 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:58:37.343669 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df1f4801-0e1d-43ee-82a1-67875b9ca2ff None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:37.343856 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df1f4801-0e1d-43ee-82a1-67875b9ca2ff None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:37.344062 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df1f4801-0e1d-43ee-82a1-67875b9ca2ff None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:37.345307 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 32/69] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:58:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:58:37.346346 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c33707-6962-4373-a25c-07758884f75a None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:37.346672 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c33707-6962-4373-a25c-07758884f75a None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:37.346980 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c33707-6962-4373-a25c-07758884f75a None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:37.348171 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 33/70] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:58:37 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:58:37.348862 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74c33707-6962-4373-a25c-07758884f75a None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:37.353750 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b93e3777-927e-4421-b099-0b686dcb37bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:37.354676 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b93e3777-927e-4421-b099-0b686dcb37bf None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:37.354999 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b93e3777-927e-4421-b099-0b686dcb37bf None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:37.355273 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b93e3777-927e-4421-b099-0b686dcb37bf None admin] PATH_INFO: `/v3/roles/member` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:37.365744 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b93e3777-927e-4421-b099-0b686dcb37bf None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:58:37.371277 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-b93e3777-927e-4421-b099-0b686dcb37bf None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 30 04:58:37.371277 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:58:37.371277 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:58:37.371277 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:58:37.371277 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:58:37.371277 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:58:37.371277 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:58:37.371277 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:58:37.371277 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:58:37.371277 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:58:37.371277 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:58:37.371277 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:58:37.371277 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:58:37.371277 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:58:37.371277 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:58:37.371277 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:58:37.372114 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:58:37.372114 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:58:37.372114 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:58:37.372114 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:58:37.372114 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:58:37.372114 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:58:37.372114 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:58:37.372114 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:58:37.372114 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:58:37.372114 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:58:37.372114 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:58:37.372114 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 04:58:37.372114 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:58:37.372114 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:58:37.372114 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:58:37.372114 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:58:37.372873 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:58:37.372873 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:58:37.372873 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:58:37.372873 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:58:37.372873 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:58:37.372873 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:58:37.372873 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:58:37.372873 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:58:37.372873 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:58:37.372873 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:58:37.372873 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 30 04:58:37.372873 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 04:58:37.372873 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 38/71] 10.222.0.77 () {56 vars in 1099 bytes} [Tue Aug 30 04:58:37 2022] GET /identity/v3/roles/member => generated 84 bytes in 245 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:58:37.373495 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b93e3777-927e-4421-b099-0b686dcb37bf None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:37.452429 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74c33707-6962-4373-a25c-07758884f75a None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:37.461345 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:37.474150 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b93e3777-927e-4421-b099-0b686dcb37bf None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:37.490767 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaa0887d-b636-4f2e-8037-060c1a05f857 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:37.491439 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaa0887d-b636-4f2e-8037-060c1a05f857 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:37.491724 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaa0887d-b636-4f2e-8037-060c1a05f857 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:37.491992 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaa0887d-b636-4f2e-8037-060c1a05f857 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:37.502073 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaa0887d-b636-4f2e-8037-060c1a05f857 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:58:37.506944 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-aaa0887d-b636-4f2e-8037-060c1a05f857 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 30 04:58:37.506944 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:58:37.506944 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:58:37.506944 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:58:37.506944 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:58:37.506944 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:58:37.506944 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:58:37.506944 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:58:37.506944 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:58:37.506944 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:58:37.506944 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:58:37.506944 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:58:37.506944 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:58:37.506944 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:58:37.506944 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:58:37.506944 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:58:37.507885 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:58:37.507885 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:58:37.507885 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:58:37.507885 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:58:37.507885 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:58:37.507885 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:58:37.507885 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:58:37.507885 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:58:37.507885 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:58:37.507885 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:58:37.507885 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:58:37.507885 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 04:58:37.507885 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:58:37.507885 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:58:37.507885 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:58:37.507885 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:58:37.508616 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:58:37.508616 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:58:37.508616 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:58:37.508616 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:58:37.508616 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:58:37.508616 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:58:37.508616 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:58:37.508616 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:58:37.508616 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:58:37.508616 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:58:37.508616 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 30 04:58:37.508616 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 04:58:37.508616 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 34/72] 10.222.0.77 () {56 vars in 1114 bytes} [Tue Aug 30 04:58:37 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:58:37.509283 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a16753d-0be2-49d0-bea2-3a8af9ea441a None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:37.509547 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a16753d-0be2-49d0-bea2-3a8af9ea441a None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:37.509739 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a16753d-0be2-49d0-bea2-3a8af9ea441a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:37.510748 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47a3cc69-bf2b-46ef-a9bf-c97816c2530c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:37.511391 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47a3cc69-bf2b-46ef-a9bf-c97816c2530c None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:37.511677 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47a3cc69-bf2b-46ef-a9bf-c97816c2530c None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:37.511950 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47a3cc69-bf2b-46ef-a9bf-c97816c2530c None admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:37.512666 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47a3cc69-bf2b-46ef-a9bf-c97816c2530c None admin] RBAC: Adding query filter params (name=admin) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:58:37.512927 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47a3cc69-bf2b-46ef-a9bf-c97816c2530c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:37.514284 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47a3cc69-bf2b-46ef-a9bf-c97816c2530c None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:37.522767 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 39/73] 10.222.0.77 () {56 vars in 1099 bytes} [Tue Aug 30 04:58:37 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:58:37.524063 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-485dafb9-4299-44cd-8adf-94071d9ae97c None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:37.524353 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-485dafb9-4299-44cd-8adf-94071d9ae97c None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:37.525152 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-485dafb9-4299-44cd-8adf-94071d9ae97c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:37.528417 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-0a16753d-0be2-49d0-bea2-3a8af9ea441a None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:58:37.529645 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-0a16753d-0be2-49d0-bea2-3a8af9ea441a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:58:37.542888 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-485dafb9-4299-44cd-8adf-94071d9ae97c None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:58:37.544058 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-485dafb9-4299-44cd-8adf-94071d9ae97c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:58:37.585659 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:37.597328 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 35/74] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:58:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:58:37.598128 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a16753d-0be2-49d0-bea2-3a8af9ea441a None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:37.620409 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 40/75] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:58:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:58:37.621745 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-485dafb9-4299-44cd-8adf-94071d9ae97c None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:37.716079 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:37.724126 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a16753d-0be2-49d0-bea2-3a8af9ea441a None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:37.740803 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-485dafb9-4299-44cd-8adf-94071d9ae97c None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:37.767123 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8034773-5ae8-4590-81e5-29e8ec54717f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:37.767979 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8034773-5ae8-4590-81e5-29e8ec54717f None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:37.768439 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8034773-5ae8-4590-81e5-29e8ec54717f None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:37.768921 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8034773-5ae8-4590-81e5-29e8ec54717f None admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:37.769761 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8034773-5ae8-4590-81e5-29e8ec54717f None admin] RBAC: Adding query filter params (name=member) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:58:37.770240 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8034773-5ae8-4590-81e5-29e8ec54717f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:37.771834 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8034773-5ae8-4590-81e5-29e8ec54717f None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:37.777944 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdef6b1e-8839-4def-bd43-f45ffd4bb1db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:37.778286 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdef6b1e-8839-4def-bd43-f45ffd4bb1db None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:37.778566 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdef6b1e-8839-4def-bd43-f45ffd4bb1db None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:37.778868 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdef6b1e-8839-4def-bd43-f45ffd4bb1db None admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:37.779533 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdef6b1e-8839-4def-bd43-f45ffd4bb1db None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:58:37.779885 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdef6b1e-8839-4def-bd43-f45ffd4bb1db None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:37.781258 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdef6b1e-8839-4def-bd43-f45ffd4bb1db None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:37.782725 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 36/76] 10.222.0.77 () {56 vars in 1101 bytes} [Tue Aug 30 04:58:37 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:58:37.783720 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8034773-5ae8-4590-81e5-29e8ec54717f None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:37.789963 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 41/77] 10.222.0.77 () {56 vars in 1111 bytes} [Tue Aug 30 04:58:37 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:58:37.790892 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdef6b1e-8839-4def-bd43-f45ffd4bb1db None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:37.839247 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:37.890053 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8034773-5ae8-4590-81e5-29e8ec54717f None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:37.896814 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdef6b1e-8839-4def-bd43-f45ffd4bb1db None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:37.925089 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f56886e0-9826-4ba8-b17e-36ff7fe0e320 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:37.925715 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f56886e0-9826-4ba8-b17e-36ff7fe0e320 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:37.925993 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f56886e0-9826-4ba8-b17e-36ff7fe0e320 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:37.926265 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f56886e0-9826-4ba8-b17e-36ff7fe0e320 None admin] PATH_INFO: `/v3/users/fdab92bac3f64739a0fcb5f729d40f85` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:37.927635 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f56886e0-9826-4ba8-b17e-36ff7fe0e320 None admin] RBAC: Authorizing `identity:get_user(user_id=fdab92bac3f64739a0fcb5f729d40f85)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:37.929090 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f56886e0-9826-4ba8-b17e-36ff7fe0e320 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:37.930319 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a7fe278-5c73-4cf6-9667-a8dfc2f644dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:37.930640 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 37/78] 10.222.0.77 () {56 vars in 1177 bytes} [Tue Aug 30 04:58:37 2022] GET /identity/v3/users/fdab92bac3f64739a0fcb5f729d40f85 => generated 247 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:58:37.930981 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a7fe278-5c73-4cf6-9667-a8dfc2f644dc None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:37.931293 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a7fe278-5c73-4cf6-9667-a8dfc2f644dc None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:37.931436 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f56886e0-9826-4ba8-b17e-36ff7fe0e320 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:37.931793 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a7fe278-5c73-4cf6-9667-a8dfc2f644dc None admin] PATH_INFO: `/v3/users/bb6b84c639b14608813a9a5e0dac6840` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:37.942875 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:37.945774 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a7fe278-5c73-4cf6-9667-a8dfc2f644dc None admin] RBAC: Authorizing `identity:get_user(user_id=bb6b84c639b14608813a9a5e0dac6840)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:37.947853 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a7fe278-5c73-4cf6-9667-a8dfc2f644dc None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:37.950034 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 42/79] 10.222.0.77 () {56 vars in 1177 bytes} [Tue Aug 30 04:58:37 2022] GET /identity/v3/users/bb6b84c639b14608813a9a5e0dac6840 => generated 275 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:58:37.951200 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a7fe278-5c73-4cf6-9667-a8dfc2f644dc None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:38.037283 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f56886e0-9826-4ba8-b17e-36ff7fe0e320 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:38.049972 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a7fe278-5c73-4cf6-9667-a8dfc2f644dc None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:38.057690 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:38.073028 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60076110-e964-4c9c-8a02-d584c242002c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:38.073655 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60076110-e964-4c9c-8a02-d584c242002c None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:38.073936 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60076110-e964-4c9c-8a02-d584c242002c None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:38.074209 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60076110-e964-4c9c-8a02-d584c242002c None admin] PATH_INFO: `/v3/users/bb6b84c639b14608813a9a5e0dac6840` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:38.075366 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60076110-e964-4c9c-8a02-d584c242002c None admin] RBAC: Authorizing `identity:get_user(user_id=bb6b84c639b14608813a9a5e0dac6840)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:38.076802 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60076110-e964-4c9c-8a02-d584c242002c None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:38.078363 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 38/80] 10.222.0.77 () {56 vars in 1177 bytes} [Tue Aug 30 04:58:37 2022] GET /identity/v3/users/bb6b84c639b14608813a9a5e0dac6840 => generated 275 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:58:38.079153 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60076110-e964-4c9c-8a02-d584c242002c None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:38.088934 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bff602df-fdd0-4d8d-afca-ec9a24692c0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:38.089558 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff602df-fdd0-4d8d-afca-ec9a24692c0d None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:38.089835 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff602df-fdd0-4d8d-afca-ec9a24692c0d None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:38.090468 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff602df-fdd0-4d8d-afca-ec9a24692c0d None admin] PATH_INFO: `/v3/roles/member` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:38.100668 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bff602df-fdd0-4d8d-afca-ec9a24692c0d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:58:38.105758 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-bff602df-fdd0-4d8d-afca-ec9a24692c0d None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 30 04:58:38.105758 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:58:38.105758 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:58:38.105758 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:58:38.105758 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:58:38.105758 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:58:38.105758 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:58:38.105758 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:58:38.105758 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:58:38.105758 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:58:38.105758 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:58:38.105758 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:58:38.105758 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:58:38.105758 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:58:38.105758 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:58:38.105758 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:58:38.106626 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:58:38.106626 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:58:38.106626 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:58:38.106626 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:58:38.106626 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:58:38.106626 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:58:38.106626 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:58:38.106626 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:58:38.106626 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:58:38.106626 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:58:38.106626 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:58:38.106626 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 04:58:38.106626 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:58:38.106626 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:58:38.106626 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:58:38.106626 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:58:38.111689 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:58:38.111689 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:58:38.111689 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:58:38.111689 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:58:38.111689 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:58:38.111689 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:58:38.111689 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:58:38.111689 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:58:38.111689 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:58:38.111689 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:58:38.111689 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 30 04:58:38.111689 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 04:58:38.111689 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 43/81] 10.222.0.77 () {56 vars in 1099 bytes} [Tue Aug 30 04:58:37 2022] GET /identity/v3/roles/member => generated 84 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:58:38.111689 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bff602df-fdd0-4d8d-afca-ec9a24692c0d None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:38.166176 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:38.176802 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60076110-e964-4c9c-8a02-d584c242002c None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:38.207564 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bff602df-fdd0-4d8d-afca-ec9a24692c0d None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:38.209683 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-768987b0-c82f-4268-a6ea-086626453c18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:38.210306 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-768987b0-c82f-4268-a6ea-086626453c18 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:38.210584 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-768987b0-c82f-4268-a6ea-086626453c18 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:38.210877 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-768987b0-c82f-4268-a6ea-086626453c18 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:38.211987 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-768987b0-c82f-4268-a6ea-086626453c18 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:38.213970 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-768987b0-c82f-4268-a6ea-086626453c18 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:38.215564 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 39/82] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 04:58:38 2022] GET /identity/v3/domains/default => generated 199 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:58:38.216327 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-768987b0-c82f-4268-a6ea-086626453c18 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:38.242345 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ef2b480-48e6-4867-9804-bb72d0f0a80d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:38.242974 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef2b480-48e6-4867-9804-bb72d0f0a80d None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:38.243255 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef2b480-48e6-4867-9804-bb72d0f0a80d None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:38.243554 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef2b480-48e6-4867-9804-bb72d0f0a80d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:38.269595 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:38.310577 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ef2b480-48e6-4867-9804-bb72d0f0a80d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:38.312085 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ef2b480-48e6-4867-9804-bb72d0f0a80d None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:38.313488 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-768987b0-c82f-4268-a6ea-086626453c18 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:38.347391 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92346577-4dd2-49d0-801d-cc6510fd430e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:38.347983 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92346577-4dd2-49d0-801d-cc6510fd430e None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:38.348263 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92346577-4dd2-49d0-801d-cc6510fd430e None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:38.348532 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92346577-4dd2-49d0-801d-cc6510fd430e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:38.368918 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:38.410616 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 44/83] 10.222.0.77 () {58 vars in 1282 bytes} [Tue Aug 30 04:58:38 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 303 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 04:58:38.411464 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ef2b480-48e6-4867-9804-bb72d0f0a80d None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:38.415922 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92346577-4dd2-49d0-801d-cc6510fd430e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:38.417438 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92346577-4dd2-49d0-801d-cc6510fd430e None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:38.470533 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:38.512832 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ef2b480-48e6-4867-9804-bb72d0f0a80d None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:38.514756 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 40/84] 10.222.0.77 () {58 vars in 1282 bytes} [Tue Aug 30 04:58:38 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 299 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 04:58:38.515614 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92346577-4dd2-49d0-801d-cc6510fd430e None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:38.546201 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c78059b-a15c-4708-80c5-acf0340a65b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:38.546799 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c78059b-a15c-4708-80c5-acf0340a65b6 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:38.547102 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c78059b-a15c-4708-80c5-acf0340a65b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:38.547380 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c78059b-a15c-4708-80c5-acf0340a65b6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:38.574696 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:38.613880 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92346577-4dd2-49d0-801d-cc6510fd430e None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:38.614749 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c78059b-a15c-4708-80c5-acf0340a65b6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:38.616249 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c78059b-a15c-4708-80c5-acf0340a65b6 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:38.647285 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fd25878-e7cc-4ead-87b3-47a9060a98fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:38.647876 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fd25878-e7cc-4ead-87b3-47a9060a98fb None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:38.648154 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fd25878-e7cc-4ead-87b3-47a9060a98fb None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:38.648423 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fd25878-e7cc-4ead-87b3-47a9060a98fb None admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:38.649074 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fd25878-e7cc-4ead-87b3-47a9060a98fb None admin] RBAC: Adding query filter params (name=member) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:58:38.649372 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fd25878-e7cc-4ead-87b3-47a9060a98fb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:38.650762 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fd25878-e7cc-4ead-87b3-47a9060a98fb None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:38.658749 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 41/85] 10.222.0.77 () {56 vars in 1101 bytes} [Tue Aug 30 04:58:38 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:58:38.659438 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fd25878-e7cc-4ead-87b3-47a9060a98fb None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:38.678032 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:38.716484 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 45/86] 10.222.0.77 () {58 vars in 1282 bytes} [Tue Aug 30 04:58:38 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 305 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 04:58:38.717313 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c78059b-a15c-4708-80c5-acf0340a65b6 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:38.758061 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fd25878-e7cc-4ead-87b3-47a9060a98fb None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:38.773358 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:38.793050 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-995a5f7d-2706-455e-bb0d-dcae5551cda4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:38.793662 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-995a5f7d-2706-455e-bb0d-dcae5551cda4 None admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:38.793945 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-995a5f7d-2706-455e-bb0d-dcae5551cda4 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:38.794220 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-995a5f7d-2706-455e-bb0d-dcae5551cda4 None admin] PATH_INFO: `/v3/users` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:38.794896 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-995a5f7d-2706-455e-bb0d-dcae5551cda4 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:38.796343 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-995a5f7d-2706-455e-bb0d-dcae5551cda4 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:38.815114 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c78059b-a15c-4708-80c5-acf0340a65b6 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:38.816549 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 42/87] 10.222.0.77 () {60 vars in 1132 bytes} [Tue Aug 30 04:58:38 2022] POST /identity/v3/users => generated 289 bytes in 157 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:58:38.817342 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-995a5f7d-2706-455e-bb0d-dcae5551cda4 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:38.854437 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f074a46c-f72b-4693-842a-c7a0d32f9fdb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:38.855462 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f074a46c-f72b-4693-842a-c7a0d32f9fdb None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:38.855925 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f074a46c-f72b-4693-842a-c7a0d32f9fdb None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:38.856199 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f074a46c-f72b-4693-842a-c7a0d32f9fdb None admin] PATH_INFO: `/v3/projects/9c3b4147cbea42e3a97c54416188a48c` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:38.857384 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f074a46c-f72b-4693-842a-c7a0d32f9fdb None admin] RBAC: Authorizing `identity:get_project(project_id=9c3b4147cbea42e3a97c54416188a48c)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:38.858950 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f074a46c-f72b-4693-842a-c7a0d32f9fdb None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:38.866163 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 46/88] 10.222.0.77 () {56 vars in 1186 bytes} [Tue Aug 30 04:58:38 2022] GET /identity/v3/projects/9c3b4147cbea42e3a97c54416188a48c => generated 298 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:58:38.866163 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f074a46c-f72b-4693-842a-c7a0d32f9fdb None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:38.907718 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:38.938905 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-995a5f7d-2706-455e-bb0d-dcae5551cda4 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:38.977247 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f074a46c-f72b-4693-842a-c7a0d32f9fdb None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:38.977807 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d4fd623-fdef-411a-aa26-48340669b84b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:38.978446 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d4fd623-fdef-411a-aa26-48340669b84b None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:38.978723 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d4fd623-fdef-411a-aa26-48340669b84b None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:38.979276 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d4fd623-fdef-411a-aa26-48340669b84b None admin] PATH_INFO: `/v3/projects/9c3b4147cbea42e3a97c54416188a48c` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:38.980425 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d4fd623-fdef-411a-aa26-48340669b84b None admin] RBAC: Authorizing `identity:get_project(project_id=9c3b4147cbea42e3a97c54416188a48c)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:38.981872 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d4fd623-fdef-411a-aa26-48340669b84b None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:38.983539 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 43/89] 10.222.0.77 () {56 vars in 1186 bytes} [Tue Aug 30 04:58:38 2022] GET /identity/v3/projects/9c3b4147cbea42e3a97c54416188a48c => generated 298 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:58:38.984296 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d4fd623-fdef-411a-aa26-48340669b84b None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:39.008959 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:39.018137 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c042b140-1833-4444-8816-c02e33331e90 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:39.018759 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c042b140-1833-4444-8816-c02e33331e90 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:39.019094 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c042b140-1833-4444-8816-c02e33331e90 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:39.019370 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c042b140-1833-4444-8816-c02e33331e90 None admin] PATH_INFO: `/v3/users/bb6b84c639b14608813a9a5e0dac6840` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:39.020598 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c042b140-1833-4444-8816-c02e33331e90 None admin] RBAC: Authorizing `identity:get_user(user_id=bb6b84c639b14608813a9a5e0dac6840)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:39.022660 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c042b140-1833-4444-8816-c02e33331e90 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:39.025434 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 47/90] 10.222.0.77 () {56 vars in 1177 bytes} [Tue Aug 30 04:58:38 2022] GET /identity/v3/users/bb6b84c639b14608813a9a5e0dac6840 => generated 275 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:58:39.025434 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c042b140-1833-4444-8816-c02e33331e90 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:39.120437 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d4fd623-fdef-411a-aa26-48340669b84b None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:39.157536 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c042b140-1833-4444-8816-c02e33331e90 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:39.163644 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e519169-2a57-467c-8bf6-b73aa8018e2c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:39.164073 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e519169-2a57-467c-8bf6-b73aa8018e2c None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:39.164351 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e519169-2a57-467c-8bf6-b73aa8018e2c None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:39.164621 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e519169-2a57-467c-8bf6-b73aa8018e2c None admin] PATH_INFO: `/v3/projects/9c3b4147cbea42e3a97c54416188a48c` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:39.166979 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e519169-2a57-467c-8bf6-b73aa8018e2c None admin] RBAC: Authorizing `identity:get_project(project_id=9c3b4147cbea42e3a97c54416188a48c)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:39.167987 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e519169-2a57-467c-8bf6-b73aa8018e2c None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:39.169811 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 44/91] 10.222.0.77 () {56 vars in 1186 bytes} [Tue Aug 30 04:58:38 2022] GET /identity/v3/projects/9c3b4147cbea42e3a97c54416188a48c => generated 298 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:58:39.170462 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e519169-2a57-467c-8bf6-b73aa8018e2c None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:39.199620 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:39.200954 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5f14bcb-372b-4ebb-a453-7dedf846f27f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:39.201573 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f14bcb-372b-4ebb-a453-7dedf846f27f None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:39.201912 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f14bcb-372b-4ebb-a453-7dedf846f27f None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:39.202193 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f14bcb-372b-4ebb-a453-7dedf846f27f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:39.203425 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5f14bcb-372b-4ebb-a453-7dedf846f27f None admin] RBAC: Adding query filter params (role.id=9d9d5d2c270146488249b7ed943280db, scope.project.id=9c3b4147cbea42e3a97c54416188a48c, user.id=fdab92bac3f64739a0fcb5f729d40f85) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:58:39.203755 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5f14bcb-372b-4ebb-a453-7dedf846f27f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:39.205477 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5f14bcb-372b-4ebb-a453-7dedf846f27f None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:39.213897 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 48/92] 10.222.0.77 () {56 vars in 1374 bytes} [Tue Aug 30 04:58:39 2022] GET /identity/v3/role_assignments?user.id=fdab92bac3f64739a0fcb5f729d40f85&scope.project.id=9c3b4147cbea42e3a97c54416188a48c&role.id=9d9d5d2c270146488249b7ed943280db => generated 259 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:58:39.214708 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5f14bcb-372b-4ebb-a453-7dedf846f27f None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:39.302214 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:39.355505 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5f14bcb-372b-4ebb-a453-7dedf846f27f None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:39.393786 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-258464e1-1f3c-4ac6-8817-d00797e6a36f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:39.394414 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-258464e1-1f3c-4ac6-8817-d00797e6a36f None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:39.394693 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-258464e1-1f3c-4ac6-8817-d00797e6a36f None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:39.394987 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-258464e1-1f3c-4ac6-8817-d00797e6a36f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:39.419334 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e519169-2a57-467c-8bf6-b73aa8018e2c None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:39.467894 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e97f0bb8-3dff-4a8b-8bf2-4d46fa0518af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:39.468570 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e97f0bb8-3dff-4a8b-8bf2-4d46fa0518af None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:39.468855 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e97f0bb8-3dff-4a8b-8bf2-4d46fa0518af None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:39.469132 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e97f0bb8-3dff-4a8b-8bf2-4d46fa0518af None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:39.469992 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e97f0bb8-3dff-4a8b-8bf2-4d46fa0518af None admin] RBAC: Adding query filter params (role.id=6f03d86980c34c36b44a8d6764ff4fda, scope.project.id=9c3b4147cbea42e3a97c54416188a48c, user.id=bb6b84c639b14608813a9a5e0dac6840) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:58:39.470308 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e97f0bb8-3dff-4a8b-8bf2-4d46fa0518af None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:39.471796 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e97f0bb8-3dff-4a8b-8bf2-4d46fa0518af None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:39.489702 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 45/93] 10.222.0.77 () {56 vars in 1374 bytes} [Tue Aug 30 04:58:39 2022] GET /identity/v3/role_assignments?user.id=bb6b84c639b14608813a9a5e0dac6840&scope.project.id=9c3b4147cbea42e3a97c54416188a48c&role.id=6f03d86980c34c36b44a8d6764ff4fda => generated 259 bytes in 319 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:58:39.490344 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e97f0bb8-3dff-4a8b-8bf2-4d46fa0518af None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:39.519540 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-258464e1-1f3c-4ac6-8817-d00797e6a36f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:39.521025 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-258464e1-1f3c-4ac6-8817-d00797e6a36f None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:39.601963 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:39.639555 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e97f0bb8-3dff-4a8b-8bf2-4d46fa0518af None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:39.685503 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d83a4aa3-62c4-4059-933e-5fd76f06571b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:39.687371 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d83a4aa3-62c4-4059-933e-5fd76f06571b None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:39.687915 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:39.689519 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d83a4aa3-62c4-4059-933e-5fd76f06571b None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:39.690505 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d83a4aa3-62c4-4059-933e-5fd76f06571b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:39.693027 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d83a4aa3-62c4-4059-933e-5fd76f06571b None admin] RBAC: Adding query filter params (role.id=3b9d5509a2f7478c9b4989d921daef2a, scope.project.id=9c3b4147cbea42e3a97c54416188a48c, user.id=bb6b84c639b14608813a9a5e0dac6840) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:58:39.693546 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d83a4aa3-62c4-4059-933e-5fd76f06571b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:39.696313 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d83a4aa3-62c4-4059-933e-5fd76f06571b None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:39.709750 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 46/94] 10.222.0.77 () {56 vars in 1374 bytes} [Tue Aug 30 04:58:39 2022] GET /identity/v3/role_assignments?user.id=bb6b84c639b14608813a9a5e0dac6840&scope.project.id=9c3b4147cbea42e3a97c54416188a48c&role.id=3b9d5509a2f7478c9b4989d921daef2a => generated 259 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:58:39.753364 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 49/95] 10.222.0.77 () {58 vars in 1282 bytes} [Tue Aug 30 04:58:39 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 539 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 04:58:39.760184 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d83a4aa3-62c4-4059-933e-5fd76f06571b None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:39.877326 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d83a4aa3-62c4-4059-933e-5fd76f06571b None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:39.919580 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00dd28b6-cb2e-410f-a63a-0ebfab54e11a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:39.920529 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00dd28b6-cb2e-410f-a63a-0ebfab54e11a None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:39.921018 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00dd28b6-cb2e-410f-a63a-0ebfab54e11a None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:39.921608 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00dd28b6-cb2e-410f-a63a-0ebfab54e11a None admin] PATH_INFO: `/v3/projects/8b38c5abf3d344cc8144125aa0a9abb1` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:39.923098 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00dd28b6-cb2e-410f-a63a-0ebfab54e11a None admin] RBAC: Authorizing `identity:get_project(project_id=8b38c5abf3d344cc8144125aa0a9abb1)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:39.924846 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00dd28b6-cb2e-410f-a63a-0ebfab54e11a None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:39.926738 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 47/96] 10.222.0.77 () {56 vars in 1186 bytes} [Tue Aug 30 04:58:39 2022] GET /identity/v3/projects/8b38c5abf3d344cc8144125aa0a9abb1 => generated 312 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:58:39.933576 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-258464e1-1f3c-4ac6-8817-d00797e6a36f None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:40.065169 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-258464e1-1f3c-4ac6-8817-d00797e6a36f None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:40.110409 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18c573af-322a-4190-a8e2-22aeaa3077e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:40.111458 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18c573af-322a-4190-a8e2-22aeaa3077e5 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:40.111922 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18c573af-322a-4190-a8e2-22aeaa3077e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:40.112396 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18c573af-322a-4190-a8e2-22aeaa3077e5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:40.113311 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18c573af-322a-4190-a8e2-22aeaa3077e5 None admin] RBAC: Adding query filter params (role.id=6f03d86980c34c36b44a8d6764ff4fda, scope.project.id=8b38c5abf3d344cc8144125aa0a9abb1, user.id=bb6b84c639b14608813a9a5e0dac6840) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:58:40.113800 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18c573af-322a-4190-a8e2-22aeaa3077e5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:40.115529 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18c573af-322a-4190-a8e2-22aeaa3077e5 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:40.122738 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 50/97] 10.222.0.77 () {56 vars in 1374 bytes} [Tue Aug 30 04:58:39 2022] GET /identity/v3/role_assignments?user.id=bb6b84c639b14608813a9a5e0dac6840&scope.project.id=8b38c5abf3d344cc8144125aa0a9abb1&role.id=6f03d86980c34c36b44a8d6764ff4fda => generated 259 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:58:41.170902 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc932914-22d7-4868-a209-958f4b02a3b4 None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:41.171407 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc932914-22d7-4868-a209-958f4b02a3b4 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:41.171480 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc932914-22d7-4868-a209-958f4b02a3b4 None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:41.172791 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 48/98] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:58:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:58:41.177858 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd97d6ef-72ef-4a07-882e-d780f85ddd54 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:41.178257 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd97d6ef-72ef-4a07-882e-d780f85ddd54 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:41.178420 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd97d6ef-72ef-4a07-882e-d780f85ddd54 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:41.197156 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-bd97d6ef-72ef-4a07-882e-d780f85ddd54 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:58:41.198331 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-bd97d6ef-72ef-4a07-882e-d780f85ddd54 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:58:41.217244 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:41.269373 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 51/99] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:58:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:58:41.619763 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc932914-22d7-4868-a209-958f4b02a3b4 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:41.630937 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:41.745887 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc932914-22d7-4868-a209-958f4b02a3b4 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:41.787578 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bab292e-d5eb-4bbd-a7b6-41d0d27c6a9f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:41.788211 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bab292e-d5eb-4bbd-a7b6-41d0d27c6a9f None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:41.788491 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bab292e-d5eb-4bbd-a7b6-41d0d27c6a9f None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:41.788764 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bab292e-d5eb-4bbd-a7b6-41d0d27c6a9f None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:41.802612 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bab292e-d5eb-4bbd-a7b6-41d0d27c6a9f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:58:41.808518 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-6bab292e-d5eb-4bbd-a7b6-41d0d27c6a9f None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 30 04:58:41.808518 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:58:41.808518 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:58:41.808518 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:58:41.808518 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:58:41.808518 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:58:41.808518 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:58:41.808518 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:58:41.808518 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:58:41.808518 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:58:41.808518 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:58:41.808518 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:58:41.808518 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:58:41.808518 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:58:41.808518 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:58:41.808518 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:58:41.809360 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:58:41.809360 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:58:41.809360 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:58:41.809360 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:58:41.809360 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:58:41.809360 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:58:41.809360 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:58:41.809360 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:58:41.809360 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:58:41.809360 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:58:41.809360 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:58:41.809360 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 04:58:41.809360 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:58:41.809360 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:58:41.809360 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:58:41.809360 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:58:41.810109 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:58:41.810109 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:58:41.810109 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:58:41.810109 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:58:41.810109 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:58:41.810109 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:58:41.810109 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:58:41.810109 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:58:41.810109 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:58:41.810109 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:58:41.810109 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 30 04:58:41.810109 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 04:58:41.810109 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 49/100] 10.222.0.77 () {56 vars in 1099 bytes} [Tue Aug 30 04:58:41 2022] GET /identity/v3/roles/reader => generated 84 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:58:41.816706 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd97d6ef-72ef-4a07-882e-d780f85ddd54 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:41.836611 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0b76dad-2693-4031-a3b7-e50ce89a1802 None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:41.836922 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0b76dad-2693-4031-a3b7-e50ce89a1802 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:41.837238 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0b76dad-2693-4031-a3b7-e50ce89a1802 None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:41.838568 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 50/101] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:58:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 04:58:41.843699 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273bcf20-7f8f-474a-8b2e-db6988ba03f8 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:41.843978 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273bcf20-7f8f-474a-8b2e-db6988ba03f8 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:41.844249 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273bcf20-7f8f-474a-8b2e-db6988ba03f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:41.871369 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-273bcf20-7f8f-474a-8b2e-db6988ba03f8 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:58:41.874549 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-273bcf20-7f8f-474a-8b2e-db6988ba03f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:58:42.003289 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:42.012386 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 51/102] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:58:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 30 04:58:42.014806 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd97d6ef-72ef-4a07-882e-d780f85ddd54 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:42.062469 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccc9c2b6-4999-461d-94f2-259bcf21ed69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:42.063112 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccc9c2b6-4999-461d-94f2-259bcf21ed69 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:42.063389 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccc9c2b6-4999-461d-94f2-259bcf21ed69 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:42.063655 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccc9c2b6-4999-461d-94f2-259bcf21ed69 None admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:42.064320 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccc9c2b6-4999-461d-94f2-259bcf21ed69 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:58:42.064617 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccc9c2b6-4999-461d-94f2-259bcf21ed69 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:42.065981 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccc9c2b6-4999-461d-94f2-259bcf21ed69 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:42.078527 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 52/103] 10.222.0.77 () {56 vars in 1101 bytes} [Tue Aug 30 04:58:41 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 262 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:58:42.084042 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-273bcf20-7f8f-474a-8b2e-db6988ba03f8 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:42.222709 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-273bcf20-7f8f-474a-8b2e-db6988ba03f8 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:42.267622 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4f9d33a-c775-4350-8095-45f1ababecac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:42.268179 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4f9d33a-c775-4350-8095-45f1ababecac None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:42.268459 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4f9d33a-c775-4350-8095-45f1ababecac None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:42.268731 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4f9d33a-c775-4350-8095-45f1ababecac None admin] PATH_INFO: `/v3/users/87dcae03963b47e68a40b6afa7553b20` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:42.287549 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4f9d33a-c775-4350-8095-45f1ababecac None admin] RBAC: Authorizing `identity:get_user(user_id=87dcae03963b47e68a40b6afa7553b20)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:42.289352 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4f9d33a-c775-4350-8095-45f1ababecac None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:42.291821 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 52/104] 10.222.0.77 () {56 vars in 1177 bytes} [Tue Aug 30 04:58:42 2022] GET /identity/v3/users/87dcae03963b47e68a40b6afa7553b20 => generated 289 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:58:42.297379 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccc9c2b6-4999-461d-94f2-259bcf21ed69 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:42.325848 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:42.442650 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccc9c2b6-4999-461d-94f2-259bcf21ed69 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:42.489658 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79bdd089-8266-4cc5-a028-a6b42185810e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:42.490277 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79bdd089-8266-4cc5-a028-a6b42185810e None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:42.490561 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79bdd089-8266-4cc5-a028-a6b42185810e None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:42.490902 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79bdd089-8266-4cc5-a028-a6b42185810e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:42.517508 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c936f1f4-19b6-475b-84e0-2c873572b908 None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:42.518196 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c936f1f4-19b6-475b-84e0-2c873572b908 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:42.519127 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c936f1f4-19b6-475b-84e0-2c873572b908 None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:42.520849 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 53/105] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:58:42 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:58:42.526801 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbeb9b0f-886f-41d3-8d7b-af7b942ce6ce None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:42.527509 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbeb9b0f-886f-41d3-8d7b-af7b942ce6ce None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:42.528005 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbeb9b0f-886f-41d3-8d7b-af7b942ce6ce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:42.560621 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-bbeb9b0f-886f-41d3-8d7b-af7b942ce6ce None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:58:42.562524 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-bbeb9b0f-886f-41d3-8d7b-af7b942ce6ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:58:42.607580 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79bdd089-8266-4cc5-a028-a6b42185810e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:42.609065 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79bdd089-8266-4cc5-a028-a6b42185810e None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:42.631160 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:42.644467 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:42.664906 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 54/106] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:58:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 30 04:58:42.666477 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbeb9b0f-886f-41d3-8d7b-af7b942ce6ce None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:42.750136 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 53/107] 10.222.0.77 () {58 vars in 1282 bytes} [Tue Aug 30 04:58:42 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 453 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 04:58:42.751586 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e86b9145-73f6-47fd-afaf-c9a16e495b81 None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:42.751875 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e86b9145-73f6-47fd-afaf-c9a16e495b81 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:42.752152 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e86b9145-73f6-47fd-afaf-c9a16e495b81 None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:42.753417 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 54/108] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:58:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:58:42.754427 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43b51f52-4d5c-4f2b-b7bf-83f09e8cb285 None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:42.754712 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43b51f52-4d5c-4f2b-b7bf-83f09e8cb285 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:42.755011 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43b51f52-4d5c-4f2b-b7bf-83f09e8cb285 None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:42.756475 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 55/109] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:58:42 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:58:42.757193 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43b51f52-4d5c-4f2b-b7bf-83f09e8cb285 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:42.778717 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbeb9b0f-886f-41d3-8d7b-af7b942ce6ce None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:42.813851 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13765202-39cb-4084-9595-49c78cd8d189 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:42.814540 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13765202-39cb-4084-9595-49c78cd8d189 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:42.814832 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13765202-39cb-4084-9595-49c78cd8d189 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:42.815125 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13765202-39cb-4084-9595-49c78cd8d189 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:42.816266 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13765202-39cb-4084-9595-49c78cd8d189 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:42.817978 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13765202-39cb-4084-9595-49c78cd8d189 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:42.819957 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 55/110] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 04:58:42 2022] GET /identity/v3/domains/default => generated 199 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:58:42.821253 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7662ac16-864b-4b89-bf7c-eff9cae99422 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:42.822091 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7662ac16-864b-4b89-bf7c-eff9cae99422 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:42.822418 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7662ac16-864b-4b89-bf7c-eff9cae99422 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:42.842170 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-7662ac16-864b-4b89-bf7c-eff9cae99422 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:58:42.843390 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-7662ac16-864b-4b89-bf7c-eff9cae99422 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:58:42.884498 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43b51f52-4d5c-4f2b-b7bf-83f09e8cb285 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:42.892533 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:42.903876 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:42.914532 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 56/111] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:58:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:58:42.915958 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb65e7e-496f-4afe-9614-81a8ad688faa None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:42.916294 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb65e7e-496f-4afe-9614-81a8ad688faa None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:42.916569 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb65e7e-496f-4afe-9614-81a8ad688faa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:42.930892 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58de7b4e-921b-4665-8ef3-25217c14af01 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:42.930892 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58de7b4e-921b-4665-8ef3-25217c14af01 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:42.930892 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58de7b4e-921b-4665-8ef3-25217c14af01 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:42.930892 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58de7b4e-921b-4665-8ef3-25217c14af01 None admin] PATH_INFO: `/v3/projects/9c3b4147cbea42e3a97c54416188a48c` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:42.934928 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58de7b4e-921b-4665-8ef3-25217c14af01 None admin] RBAC: Authorizing `identity:get_project(project_id=9c3b4147cbea42e3a97c54416188a48c)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:42.934928 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58de7b4e-921b-4665-8ef3-25217c14af01 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:42.935092 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-1fb65e7e-496f-4afe-9614-81a8ad688faa None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:58:42.935160 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 56/112] 10.222.0.77 () {56 vars in 1186 bytes} [Tue Aug 30 04:58:42 2022] GET /identity/v3/projects/9c3b4147cbea42e3a97c54416188a48c => generated 298 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:58:42.935689 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-1fb65e7e-496f-4afe-9614-81a8ad688faa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:58:42.935903 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58de7b4e-921b-4665-8ef3-25217c14af01 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:43.008227 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 57/113] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:58:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:58:43.010874 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d868cb17-3947-4146-b115-7d0cf4c82e94 None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:43.010874 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d868cb17-3947-4146-b115-7d0cf4c82e94 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:43.010874 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d868cb17-3947-4146-b115-7d0cf4c82e94 None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:43.011750 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 58/114] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:58:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:58:43.012443 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d868cb17-3947-4146-b115-7d0cf4c82e94 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:43.055334 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58de7b4e-921b-4665-8ef3-25217c14af01 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:43.093461 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da6e3b4c-00df-4e95-8ca0-ca034a2a72ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:43.094314 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da6e3b4c-00df-4e95-8ca0-ca034a2a72ac None admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:43.094742 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da6e3b4c-00df-4e95-8ca0-ca034a2a72ac None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:43.095183 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da6e3b4c-00df-4e95-8ca0-ca034a2a72ac None admin] PATH_INFO: `/v3/projects` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:43.095970 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da6e3b4c-00df-4e95-8ca0-ca034a2a72ac None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:43.097552 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da6e3b4c-00df-4e95-8ca0-ca034a2a72ac None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:43.124923 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d868cb17-3947-4146-b115-7d0cf4c82e94 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:43.134004 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 57/115] 10.222.0.77 () {60 vars in 1141 bytes} [Tue Aug 30 04:58:42 2022] POST /identity/v3/projects => generated 302 bytes in 198 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:58:43.136310 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da6e3b4c-00df-4e95-8ca0-ca034a2a72ac None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:43.140538 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:43.187492 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:43.189034 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bb000fb-c91a-4fcd-9bc0-f4719f8c5e68 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:43.189034 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb000fb-c91a-4fcd-9bc0-f4719f8c5e68 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:43.189330 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb000fb-c91a-4fcd-9bc0-f4719f8c5e68 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:43.189467 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb000fb-c91a-4fcd-9bc0-f4719f8c5e68 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:43.190218 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bb000fb-c91a-4fcd-9bc0-f4719f8c5e68 None admin] RBAC: Adding query filter params (role.id=ddaf2963f700481eb93adea7a36f4e73, scope.project.id=9c3b4147cbea42e3a97c54416188a48c, user.id=87dcae03963b47e68a40b6afa7553b20) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:58:43.190524 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bb000fb-c91a-4fcd-9bc0-f4719f8c5e68 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:43.192285 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bb000fb-c91a-4fcd-9bc0-f4719f8c5e68 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:43.200824 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 59/116] 10.222.0.77 () {56 vars in 1374 bytes} [Tue Aug 30 04:58:43 2022] GET /identity/v3/role_assignments?user.id=87dcae03963b47e68a40b6afa7553b20&scope.project.id=9c3b4147cbea42e3a97c54416188a48c&role.id=ddaf2963f700481eb93adea7a36f4e73 => generated 259 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:58:43.208897 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdda0af5-dce8-4926-a897-106a54625588 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:43.209410 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdda0af5-dce8-4926-a897-106a54625588 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:43.209931 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdda0af5-dce8-4926-a897-106a54625588 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:43.246700 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-fdda0af5-dce8-4926-a897-106a54625588 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:58:43.248896 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-fdda0af5-dce8-4926-a897-106a54625588 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:58:43.278490 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da6e3b4c-00df-4e95-8ca0-ca034a2a72ac None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:43.328190 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63077511-5289-4d21-8113-80b09faf37f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:43.328794 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63077511-5289-4d21-8113-80b09faf37f4 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:43.329024 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63077511-5289-4d21-8113-80b09faf37f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:43.329307 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63077511-5289-4d21-8113-80b09faf37f4 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:43.343564 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63077511-5289-4d21-8113-80b09faf37f4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:58:43.350039 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-63077511-5289-4d21-8113-80b09faf37f4 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 30 04:58:43.350039 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:58:43.350039 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:58:43.350039 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:58:43.350039 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:58:43.350039 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:58:43.350039 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:58:43.350039 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:58:43.350039 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:58:43.350039 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:58:43.350039 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:58:43.350039 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:58:43.350039 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:58:43.350039 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:58:43.350039 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:58:43.350039 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:58:43.350974 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:58:43.350974 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:58:43.350974 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:58:43.350974 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:58:43.350974 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:58:43.350974 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:58:43.350974 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:58:43.350974 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:58:43.350974 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:58:43.350974 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:58:43.350974 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:58:43.350974 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 04:58:43.350974 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:58:43.350974 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:58:43.350974 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:58:43.350974 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:58:43.351738 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:58:43.351738 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:58:43.351738 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:58:43.351738 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:58:43.351738 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:58:43.351738 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:58:43.351738 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:58:43.351738 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:58:43.351738 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:58:43.351738 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:58:43.351738 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 30 04:58:43.351738 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 04:58:43.351738 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 58/117] 10.222.0.77 () {56 vars in 1114 bytes} [Tue Aug 30 04:58:43 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:58:43.352366 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63077511-5289-4d21-8113-80b09faf37f4 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:43.356730 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 60/118] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:58:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:58:43.357516 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdda0af5-dce8-4926-a897-106a54625588 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:43.460299 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:43.468082 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63077511-5289-4d21-8113-80b09faf37f4 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:43.479470 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdda0af5-dce8-4926-a897-106a54625588 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:43.487822 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:43.515745 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f33e8a4-aae3-4996-b4c2-07b4412cd179 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:43.516391 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f33e8a4-aae3-4996-b4c2-07b4412cd179 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:43.516667 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f33e8a4-aae3-4996-b4c2-07b4412cd179 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:43.516946 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f33e8a4-aae3-4996-b4c2-07b4412cd179 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:43.518351 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dabebb6f-cb1f-4928-8f12-177a3adbef3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:43.519268 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dabebb6f-cb1f-4928-8f12-177a3adbef3b None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:43.519747 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dabebb6f-cb1f-4928-8f12-177a3adbef3b None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:43.520171 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dabebb6f-cb1f-4928-8f12-177a3adbef3b None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:43.529877 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f33e8a4-aae3-4996-b4c2-07b4412cd179 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:58:43.531632 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dabebb6f-cb1f-4928-8f12-177a3adbef3b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:58:43.535250 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-9f33e8a4-aae3-4996-b4c2-07b4412cd179 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 30 04:58:43.535250 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:58:43.535250 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:58:43.535250 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:58:43.535250 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:58:43.535250 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:58:43.535250 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:58:43.535250 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:58:43.535250 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:58:43.535250 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:58:43.535250 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:58:43.535250 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:58:43.535250 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:58:43.535250 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:58:43.535250 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:58:43.535250 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:58:43.536259 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:58:43.536259 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:58:43.536259 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:58:43.536259 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:58:43.536259 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:58:43.536259 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:58:43.536259 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:58:43.536259 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:58:43.536259 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:58:43.536259 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:58:43.536259 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:58:43.536259 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 04:58:43.536259 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:58:43.536259 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:58:43.536259 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:58:43.536259 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:58:43.537151 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:58:43.537151 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:58:43.537151 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:58:43.537151 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:58:43.537151 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:58:43.537151 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:58:43.537151 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:58:43.537151 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:58:43.537151 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:58:43.537151 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:58:43.537151 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 30 04:58:43.537151 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 04:58:43.537151 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 59/119] 10.222.0.77 () {56 vars in 1099 bytes} [Tue Aug 30 04:58:43 2022] GET /identity/v3/roles/member => generated 84 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:58:43.537151 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f33e8a4-aae3-4996-b4c2-07b4412cd179 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:43.540014 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-dabebb6f-cb1f-4928-8f12-177a3adbef3b None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 04:58:43.540014 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:58:43.540014 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:58:43.540014 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:58:43.540014 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:58:43.540014 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:58:43.540014 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:58:43.540014 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:58:43.540014 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:58:43.540014 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:58:43.540014 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:58:43.540014 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:58:43.540014 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:58:43.540014 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:58:43.540014 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:58:43.540014 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:58:43.541343 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:58:43.541343 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:58:43.541343 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:58:43.541343 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:58:43.541343 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:58:43.541343 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:58:43.541343 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:58:43.541343 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:58:43.541343 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:58:43.541343 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:58:43.541343 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:58:43.541343 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 04:58:43.541343 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:58:43.541343 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:58:43.541343 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:58:43.541343 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:58:43.542081 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:58:43.542081 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:58:43.542081 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:58:43.542081 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:58:43.542081 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:58:43.542081 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:58:43.542081 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:58:43.542081 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:58:43.542081 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:58:43.542081 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:58:43.542081 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 04:58:43.542081 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 04:58:43.544509 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 61/120] 10.222.0.77 () {56 vars in 1096 bytes} [Tue Aug 30 04:58:43 2022] GET /identity/v3/roles/admin => generated 83 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:58:43.550273 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dabebb6f-cb1f-4928-8f12-177a3adbef3b None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:43.687430 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dabebb6f-cb1f-4928-8f12-177a3adbef3b None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:43.716867 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f33e8a4-aae3-4996-b4c2-07b4412cd179 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:43.747826 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8a059db-1535-4183-a757-f655477eb60f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:43.748791 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8a059db-1535-4183-a757-f655477eb60f None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:43.749074 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8a059db-1535-4183-a757-f655477eb60f None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:43.749346 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8a059db-1535-4183-a757-f655477eb60f None admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:43.750000 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8a059db-1535-4183-a757-f655477eb60f None admin] RBAC: Adding query filter params (name=member) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:58:43.750306 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8a059db-1535-4183-a757-f655477eb60f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:43.751786 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8a059db-1535-4183-a757-f655477eb60f None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:43.761483 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 62/121] 10.222.0.77 () {56 vars in 1101 bytes} [Tue Aug 30 04:58:43 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:58:43.762430 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8a059db-1535-4183-a757-f655477eb60f None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:43.767236 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2649a444-b7e0-4855-b46c-7e904bae7d79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:43.767868 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2649a444-b7e0-4855-b46c-7e904bae7d79 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:43.768149 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2649a444-b7e0-4855-b46c-7e904bae7d79 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:43.768423 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2649a444-b7e0-4855-b46c-7e904bae7d79 None admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:43.769086 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2649a444-b7e0-4855-b46c-7e904bae7d79 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:58:43.769388 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2649a444-b7e0-4855-b46c-7e904bae7d79 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:43.770778 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2649a444-b7e0-4855-b46c-7e904bae7d79 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:43.786565 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 60/122] 10.222.0.77 () {56 vars in 1111 bytes} [Tue Aug 30 04:58:43 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 250 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:58:43.787840 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2649a444-b7e0-4855-b46c-7e904bae7d79 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:43.803336 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:43.818335 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:43.906847 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8a059db-1535-4183-a757-f655477eb60f None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:43.910704 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2649a444-b7e0-4855-b46c-7e904bae7d79 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:43.946029 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3379b23c-b643-4afc-8e30-eeca2bd82ab4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:43.946744 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3379b23c-b643-4afc-8e30-eeca2bd82ab4 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:43.947135 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3379b23c-b643-4afc-8e30-eeca2bd82ab4 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:43.947626 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3379b23c-b643-4afc-8e30-eeca2bd82ab4 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:43.953784 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb22c72e-0e4e-4050-8cc4-b0cf2f5a036a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:43.954886 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb22c72e-0e4e-4050-8cc4-b0cf2f5a036a None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:43.955358 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb22c72e-0e4e-4050-8cc4-b0cf2f5a036a None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:43.956172 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb22c72e-0e4e-4050-8cc4-b0cf2f5a036a None admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:43.957007 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb22c72e-0e4e-4050-8cc4-b0cf2f5a036a None admin] RBAC: Adding query filter params (name=admin) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:58:43.957487 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb22c72e-0e4e-4050-8cc4-b0cf2f5a036a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:43.958674 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3379b23c-b643-4afc-8e30-eeca2bd82ab4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:58:43.959557 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb22c72e-0e4e-4050-8cc4-b0cf2f5a036a None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:43.964797 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-3379b23c-b643-4afc-8e30-eeca2bd82ab4 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 30 04:58:43.964797 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:58:43.964797 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:58:43.964797 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:58:43.964797 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:58:43.964797 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:58:43.964797 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:58:43.964797 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:58:43.964797 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:58:43.964797 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:58:43.964797 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:58:43.964797 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:58:43.964797 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:58:43.964797 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:58:43.964797 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:58:43.964797 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:58:43.965635 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:58:43.965635 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:58:43.965635 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:58:43.965635 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:58:43.965635 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:58:43.965635 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:58:43.965635 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:58:43.965635 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:58:43.965635 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:58:43.965635 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:58:43.965635 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:58:43.965635 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 04:58:43.965635 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:58:43.965635 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:58:43.965635 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:58:43.965635 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:58:43.966587 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:58:43.966587 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:58:43.966587 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:58:43.966587 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:58:43.966587 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:58:43.966587 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:58:43.966587 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:58:43.966587 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:58:43.966587 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:58:43.966587 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:58:43.966587 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 30 04:58:43.966587 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 04:58:43.966587 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 61/123] 10.222.0.77 () {56 vars in 1099 bytes} [Tue Aug 30 04:58:43 2022] GET /identity/v3/roles/member => generated 84 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:58:43.967430 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3379b23c-b643-4afc-8e30-eeca2bd82ab4 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:43.976679 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 63/124] 10.222.0.77 () {56 vars in 1099 bytes} [Tue Aug 30 04:58:43 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:58:43.976679 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb22c72e-0e4e-4050-8cc4-b0cf2f5a036a None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:44.059849 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:44.084099 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3379b23c-b643-4afc-8e30-eeca2bd82ab4 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:44.105256 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:44.119123 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb22c72e-0e4e-4050-8cc4-b0cf2f5a036a None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:44.120881 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1277ee1-c20f-45fc-9f5d-ce5ac6d97ba8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:44.121699 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1277ee1-c20f-45fc-9f5d-ce5ac6d97ba8 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:44.121984 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1277ee1-c20f-45fc-9f5d-ce5ac6d97ba8 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:44.122307 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1277ee1-c20f-45fc-9f5d-ce5ac6d97ba8 None admin] PATH_INFO: `/v3/users/bb6b84c639b14608813a9a5e0dac6840` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:44.123526 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1277ee1-c20f-45fc-9f5d-ce5ac6d97ba8 None admin] RBAC: Authorizing `identity:get_user(user_id=bb6b84c639b14608813a9a5e0dac6840)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:44.124995 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1277ee1-c20f-45fc-9f5d-ce5ac6d97ba8 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:44.126759 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 62/125] 10.222.0.77 () {56 vars in 1177 bytes} [Tue Aug 30 04:58:43 2022] GET /identity/v3/users/bb6b84c639b14608813a9a5e0dac6840 => generated 275 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:58:44.131854 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1277ee1-c20f-45fc-9f5d-ce5ac6d97ba8 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:44.163908 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f5bb57b-340d-4ab4-b05b-ea6da2940bdc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:44.165123 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f5bb57b-340d-4ab4-b05b-ea6da2940bdc None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:44.166496 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f5bb57b-340d-4ab4-b05b-ea6da2940bdc None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:44.167254 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f5bb57b-340d-4ab4-b05b-ea6da2940bdc None admin] PATH_INFO: `/v3/users/bb6b84c639b14608813a9a5e0dac6840` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:44.168597 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f5bb57b-340d-4ab4-b05b-ea6da2940bdc None admin] RBAC: Authorizing `identity:get_user(user_id=bb6b84c639b14608813a9a5e0dac6840)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:44.171336 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f5bb57b-340d-4ab4-b05b-ea6da2940bdc None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:44.173453 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 64/126] 10.222.0.77 () {56 vars in 1177 bytes} [Tue Aug 30 04:58:43 2022] GET /identity/v3/users/bb6b84c639b14608813a9a5e0dac6840 => generated 275 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:58:44.175044 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f5bb57b-340d-4ab4-b05b-ea6da2940bdc None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:44.239081 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1277ee1-c20f-45fc-9f5d-ce5ac6d97ba8 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:44.275557 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-923836de-e5a6-4508-959c-f64e5e239be6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:44.276162 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-923836de-e5a6-4508-959c-f64e5e239be6 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:44.276443 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-923836de-e5a6-4508-959c-f64e5e239be6 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:44.276714 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-923836de-e5a6-4508-959c-f64e5e239be6 None admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:44.277368 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-923836de-e5a6-4508-959c-f64e5e239be6 None admin] RBAC: Adding query filter params (name=member) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:58:44.277668 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-923836de-e5a6-4508-959c-f64e5e239be6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:44.279090 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-923836de-e5a6-4508-959c-f64e5e239be6 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:44.287931 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 63/127] 10.222.0.77 () {56 vars in 1101 bytes} [Tue Aug 30 04:58:44 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:58:44.288656 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-923836de-e5a6-4508-959c-f64e5e239be6 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:44.300304 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:44.314334 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f5bb57b-340d-4ab4-b05b-ea6da2940bdc None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:44.364366 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f2c3e06-503a-43e1-a596-f6bca0d88db3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:44.366231 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f2c3e06-503a-43e1-a596-f6bca0d88db3 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:44.366711 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f2c3e06-503a-43e1-a596-f6bca0d88db3 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:44.367219 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f2c3e06-503a-43e1-a596-f6bca0d88db3 None admin] PATH_INFO: `/v3/users/fdab92bac3f64739a0fcb5f729d40f85` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:44.368619 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f2c3e06-503a-43e1-a596-f6bca0d88db3 None admin] RBAC: Authorizing `identity:get_user(user_id=fdab92bac3f64739a0fcb5f729d40f85)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:44.370779 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f2c3e06-503a-43e1-a596-f6bca0d88db3 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:44.377845 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 65/128] 10.222.0.77 () {56 vars in 1177 bytes} [Tue Aug 30 04:58:44 2022] GET /identity/v3/users/fdab92bac3f64739a0fcb5f729d40f85 => generated 247 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:58:44.377845 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f2c3e06-503a-43e1-a596-f6bca0d88db3 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:44.403455 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-923836de-e5a6-4508-959c-f64e5e239be6 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:44.417562 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:44.445364 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca21d8d1-9b5f-42c6-a075-686558b5813a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:44.446033 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca21d8d1-9b5f-42c6-a075-686558b5813a None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:44.446318 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca21d8d1-9b5f-42c6-a075-686558b5813a None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:44.447156 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca21d8d1-9b5f-42c6-a075-686558b5813a None admin] PATH_INFO: `/v3/projects/8b38c5abf3d344cc8144125aa0a9abb1` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:44.448367 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca21d8d1-9b5f-42c6-a075-686558b5813a None admin] RBAC: Authorizing `identity:get_project(project_id=8b38c5abf3d344cc8144125aa0a9abb1)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:44.449924 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca21d8d1-9b5f-42c6-a075-686558b5813a None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:44.451913 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 64/129] 10.222.0.77 () {56 vars in 1186 bytes} [Tue Aug 30 04:58:44 2022] GET /identity/v3/projects/8b38c5abf3d344cc8144125aa0a9abb1 => generated 312 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:58:44.457715 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca21d8d1-9b5f-42c6-a075-686558b5813a None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:44.517900 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f2c3e06-503a-43e1-a596-f6bca0d88db3 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:44.553202 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:44.565843 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e92598a-b5f4-4f79-9442-f4fb2808c147 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:44.566736 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e92598a-b5f4-4f79-9442-f4fb2808c147 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:44.567686 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e92598a-b5f4-4f79-9442-f4fb2808c147 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:44.568156 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e92598a-b5f4-4f79-9442-f4fb2808c147 None admin] PATH_INFO: `/v3/projects/9c3b4147cbea42e3a97c54416188a48c` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:44.569967 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e92598a-b5f4-4f79-9442-f4fb2808c147 None admin] RBAC: Authorizing `identity:get_project(project_id=9c3b4147cbea42e3a97c54416188a48c)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:44.572934 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e92598a-b5f4-4f79-9442-f4fb2808c147 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:44.576045 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 66/130] 10.222.0.77 () {56 vars in 1186 bytes} [Tue Aug 30 04:58:44 2022] GET /identity/v3/projects/9c3b4147cbea42e3a97c54416188a48c => generated 298 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:58:44.577716 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e92598a-b5f4-4f79-9442-f4fb2808c147 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:44.580133 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca21d8d1-9b5f-42c6-a075-686558b5813a None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:44.626608 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a13e0a76-4fcd-4345-a6cc-fce44a9821a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:44.627320 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a13e0a76-4fcd-4345-a6cc-fce44a9821a5 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:44.627565 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a13e0a76-4fcd-4345-a6cc-fce44a9821a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:44.627843 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a13e0a76-4fcd-4345-a6cc-fce44a9821a5 None admin] PATH_INFO: `/v3/users/bb6b84c639b14608813a9a5e0dac6840` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:44.629059 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a13e0a76-4fcd-4345-a6cc-fce44a9821a5 None admin] RBAC: Authorizing `identity:get_user(user_id=bb6b84c639b14608813a9a5e0dac6840)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:44.630575 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a13e0a76-4fcd-4345-a6cc-fce44a9821a5 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:44.632187 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 65/131] 10.222.0.77 () {56 vars in 1177 bytes} [Tue Aug 30 04:58:44 2022] GET /identity/v3/users/bb6b84c639b14608813a9a5e0dac6840 => generated 275 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:58:44.632887 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a13e0a76-4fcd-4345-a6cc-fce44a9821a5 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:44.696051 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:44.706981 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e92598a-b5f4-4f79-9442-f4fb2808c147 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:44.744138 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a13e0a76-4fcd-4345-a6cc-fce44a9821a5 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:44.748820 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4240f5e-138b-475f-81a7-2a2c20f3309f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:44.749562 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4240f5e-138b-475f-81a7-2a2c20f3309f None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:44.749949 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4240f5e-138b-475f-81a7-2a2c20f3309f None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:44.750328 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4240f5e-138b-475f-81a7-2a2c20f3309f None admin] PATH_INFO: `/v3/projects/9c3b4147cbea42e3a97c54416188a48c` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:44.751645 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4240f5e-138b-475f-81a7-2a2c20f3309f None admin] RBAC: Authorizing `identity:get_project(project_id=9c3b4147cbea42e3a97c54416188a48c)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:44.753569 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4240f5e-138b-475f-81a7-2a2c20f3309f None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:44.759844 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 67/132] 10.222.0.77 () {56 vars in 1186 bytes} [Tue Aug 30 04:58:44 2022] GET /identity/v3/projects/9c3b4147cbea42e3a97c54416188a48c => generated 298 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:58:44.759844 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4240f5e-138b-475f-81a7-2a2c20f3309f None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:44.795126 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec44ec68-e746-407c-817f-ac2dca399ced None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:44.797208 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec44ec68-e746-407c-817f-ac2dca399ced None admin] REQUEST_METHOD: `PUT` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:44.797696 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec44ec68-e746-407c-817f-ac2dca399ced None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:44.798165 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec44ec68-e746-407c-817f-ac2dca399ced None admin] PATH_INFO: `/v3/projects/8b38c5abf3d344cc8144125aa0a9abb1/users/bb6b84c639b14608813a9a5e0dac6840/roles/6f03d86980c34c36b44a8d6764ff4fda` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:44.800367 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec44ec68-e746-407c-817f-ac2dca399ced None admin] RBAC: Authorizing `identity:create_grant(project_id=8b38c5abf3d344cc8144125aa0a9abb1, user_id=bb6b84c639b14608813a9a5e0dac6840, role_id=6f03d86980c34c36b44a8d6764ff4fda)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:44.802525 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec44ec68-e746-407c-817f-ac2dca399ced None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:44.820924 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 66/133] 10.222.0.77 () {58 vars in 1439 bytes} [Tue Aug 30 04:58:44 2022] PUT /identity/v3/projects/8b38c5abf3d344cc8144125aa0a9abb1/users/bb6b84c639b14608813a9a5e0dac6840/roles/6f03d86980c34c36b44a8d6764ff4fda => generated 0 bytes in 188 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:58:44.823264 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec44ec68-e746-407c-817f-ac2dca399ced None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:44.829110 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:44.878059 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4240f5e-138b-475f-81a7-2a2c20f3309f None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:44.926435 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-884e6989-ddde-462e-b2d9-78e7f36eb476 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:44.927834 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-884e6989-ddde-462e-b2d9-78e7f36eb476 None admin] REQUEST_METHOD: `PUT` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:44.927834 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-884e6989-ddde-462e-b2d9-78e7f36eb476 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:44.927995 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-884e6989-ddde-462e-b2d9-78e7f36eb476 None admin] PATH_INFO: `/v3/projects/9c3b4147cbea42e3a97c54416188a48c/users/bb6b84c639b14608813a9a5e0dac6840/roles/3b9d5509a2f7478c9b4989d921daef2a` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:44.930281 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-884e6989-ddde-462e-b2d9-78e7f36eb476 None admin] RBAC: Authorizing `identity:create_grant(project_id=9c3b4147cbea42e3a97c54416188a48c, user_id=bb6b84c639b14608813a9a5e0dac6840, role_id=3b9d5509a2f7478c9b4989d921daef2a)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:44.931933 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-884e6989-ddde-462e-b2d9-78e7f36eb476 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:44.955654 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 68/134] 10.222.0.77 () {58 vars in 1439 bytes} [Tue Aug 30 04:58:44 2022] PUT /identity/v3/projects/9c3b4147cbea42e3a97c54416188a48c/users/bb6b84c639b14608813a9a5e0dac6840/roles/3b9d5509a2f7478c9b4989d921daef2a => generated 0 bytes in 200 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:58:44.957013 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-884e6989-ddde-462e-b2d9-78e7f36eb476 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:44.973693 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec44ec68-e746-407c-817f-ac2dca399ced None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:45.029146 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:45.055791 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79998ba7-289c-43f1-8fe1-929c9c6c2f90 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:45.056412 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79998ba7-289c-43f1-8fe1-929c9c6c2f90 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:45.056695 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79998ba7-289c-43f1-8fe1-929c9c6c2f90 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:45.056971 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79998ba7-289c-43f1-8fe1-929c9c6c2f90 None admin] PATH_INFO: `/v3/projects/9c3b4147cbea42e3a97c54416188a48c` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:45.060320 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79998ba7-289c-43f1-8fe1-929c9c6c2f90 None admin] RBAC: Authorizing `identity:get_project(project_id=9c3b4147cbea42e3a97c54416188a48c)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:45.061843 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79998ba7-289c-43f1-8fe1-929c9c6c2f90 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:45.063829 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 67/135] 10.222.0.77 () {56 vars in 1186 bytes} [Tue Aug 30 04:58:44 2022] GET /identity/v3/projects/9c3b4147cbea42e3a97c54416188a48c => generated 298 bytes in 241 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:58:45.070527 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79998ba7-289c-43f1-8fe1-929c9c6c2f90 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:45.159216 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:45.186787 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79998ba7-289c-43f1-8fe1-929c9c6c2f90 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:45.224888 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c99b4ec5-4361-46a9-a407-d401f231f13d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:45.225613 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c99b4ec5-4361-46a9-a407-d401f231f13d None admin] REQUEST_METHOD: `PUT` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:45.225896 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c99b4ec5-4361-46a9-a407-d401f231f13d None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:45.226219 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c99b4ec5-4361-46a9-a407-d401f231f13d None admin] PATH_INFO: `/v3/projects/9c3b4147cbea42e3a97c54416188a48c/users/bb6b84c639b14608813a9a5e0dac6840/roles/6f03d86980c34c36b44a8d6764ff4fda` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:45.228232 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c99b4ec5-4361-46a9-a407-d401f231f13d None admin] RBAC: Authorizing `identity:create_grant(project_id=9c3b4147cbea42e3a97c54416188a48c, user_id=bb6b84c639b14608813a9a5e0dac6840, role_id=6f03d86980c34c36b44a8d6764ff4fda)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:45.229952 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c99b4ec5-4361-46a9-a407-d401f231f13d None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:45.247285 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-884e6989-ddde-462e-b2d9-78e7f36eb476 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:45.248281 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 68/136] 10.222.0.77 () {58 vars in 1439 bytes} [Tue Aug 30 04:58:45 2022] PUT /identity/v3/projects/9c3b4147cbea42e3a97c54416188a48c/users/bb6b84c639b14608813a9a5e0dac6840/roles/6f03d86980c34c36b44a8d6764ff4fda => generated 0 bytes in 178 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 04:58:45.307997 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55e713ee-4d91-49ce-b0c3-02de7cfb63f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:45.308643 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55e713ee-4d91-49ce-b0c3-02de7cfb63f6 None admin] REQUEST_METHOD: `PUT` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:45.308928 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55e713ee-4d91-49ce-b0c3-02de7cfb63f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:45.309199 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55e713ee-4d91-49ce-b0c3-02de7cfb63f6 None admin] PATH_INFO: `/v3/projects/9c3b4147cbea42e3a97c54416188a48c/users/fdab92bac3f64739a0fcb5f729d40f85/roles/9d9d5d2c270146488249b7ed943280db` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:45.312683 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55e713ee-4d91-49ce-b0c3-02de7cfb63f6 None admin] RBAC: Authorizing `identity:create_grant(project_id=9c3b4147cbea42e3a97c54416188a48c, user_id=fdab92bac3f64739a0fcb5f729d40f85, role_id=9d9d5d2c270146488249b7ed943280db)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:45.314204 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55e713ee-4d91-49ce-b0c3-02de7cfb63f6 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:45.333588 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 69/137] 10.222.0.77 () {58 vars in 1439 bytes} [Tue Aug 30 04:58:44 2022] PUT /identity/v3/projects/9c3b4147cbea42e3a97c54416188a48c/users/fdab92bac3f64739a0fcb5f729d40f85/roles/9d9d5d2c270146488249b7ed943280db => generated 0 bytes in 377 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:58:45.521904 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-631f85d6-33fc-4b1b-9ef7-793201c193a6 None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:45.522407 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-631f85d6-33fc-4b1b-9ef7-793201c193a6 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:45.522479 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-631f85d6-33fc-4b1b-9ef7-793201c193a6 None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:45.523845 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 69/138] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:58:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:58:45.529562 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f1b926a-64eb-4cec-b029-8046b90573ef None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:45.530193 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f1b926a-64eb-4cec-b029-8046b90573ef None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:45.530671 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f1b926a-64eb-4cec-b029-8046b90573ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:45.536175 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd87e620-0541-4c05-9f21-5ed83f30662a None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:45.536741 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd87e620-0541-4c05-9f21-5ed83f30662a None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:45.537279 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd87e620-0541-4c05-9f21-5ed83f30662a None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:45.538925 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 70/139] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:58:45 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 04:58:45.544592 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6480117c-ca67-441c-8167-81694c33cf9c None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:45.545156 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6480117c-ca67-441c-8167-81694c33cf9c None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:45.545633 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6480117c-ca67-441c-8167-81694c33cf9c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:45.572098 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-8f1b926a-64eb-4cec-b029-8046b90573ef None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:58:45.575124 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-8f1b926a-64eb-4cec-b029-8046b90573ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:58:45.581782 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-6480117c-ca67-441c-8167-81694c33cf9c None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:58:45.584269 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-6480117c-ca67-441c-8167-81694c33cf9c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:58:45.675761 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 70/140] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:58:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:58:45.677888 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 71/141] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:58:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 30 04:58:46.046866 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f1b926a-64eb-4cec-b029-8046b90573ef None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:46.069886 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:46.098605 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6480117c-ca67-441c-8167-81694c33cf9c None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:46.193731 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:46.226905 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f1b926a-64eb-4cec-b029-8046b90573ef None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:46.247904 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6480117c-ca67-441c-8167-81694c33cf9c None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:46.305333 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d242380-feae-40e5-9d92-587de358de7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:46.306326 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d242380-feae-40e5-9d92-587de358de7d None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:46.307430 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d242380-feae-40e5-9d92-587de358de7d None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:46.308118 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d242380-feae-40e5-9d92-587de358de7d None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:46.322287 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a282f05-f904-4f9f-9e21-f5fe95a14eae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:46.323663 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a282f05-f904-4f9f-9e21-f5fe95a14eae None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:46.324640 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a282f05-f904-4f9f-9e21-f5fe95a14eae None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:46.325240 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a282f05-f904-4f9f-9e21-f5fe95a14eae None admin] PATH_INFO: `/v3/domains/default` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:46.328749 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a282f05-f904-4f9f-9e21-f5fe95a14eae None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:46.330492 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a282f05-f904-4f9f-9e21-f5fe95a14eae None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:46.331227 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d242380-feae-40e5-9d92-587de358de7d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:58:46.336309 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-3d242380-feae-40e5-9d92-587de358de7d None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 30 04:58:46.336309 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:58:46.336309 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:58:46.336309 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:58:46.336309 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:58:46.336309 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:58:46.336309 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:58:46.336309 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:58:46.336309 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:58:46.336309 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:58:46.336309 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:58:46.336309 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:58:46.336309 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:58:46.336309 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:58:46.336309 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:58:46.336309 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:58:46.338105 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:58:46.338105 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:58:46.338105 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:58:46.338105 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:58:46.338105 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:58:46.338105 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:58:46.338105 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:58:46.338105 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:58:46.338105 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:58:46.338105 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:58:46.338105 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:58:46.338105 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 04:58:46.338105 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:58:46.338105 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:58:46.338105 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:58:46.338105 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:58:46.339255 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:58:46.339255 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:58:46.339255 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:58:46.339255 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:58:46.339255 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:58:46.339255 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:58:46.339255 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:58:46.339255 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:58:46.339255 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:58:46.339255 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:58:46.339255 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 30 04:58:46.339255 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 04:58:46.340624 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 72/142] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 04:58:46 2022] GET /identity/v3/domains/default => generated 199 bytes in 243 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:58:46.341468 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 71/143] 10.222.0.77 () {56 vars in 1099 bytes} [Tue Aug 30 04:58:46 2022] GET /identity/v3/roles/reader => generated 84 bytes in 295 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:58:46.346762 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a282f05-f904-4f9f-9e21-f5fe95a14eae None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:46.352063 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d242380-feae-40e5-9d92-587de358de7d None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:46.438009 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:46.529107 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d242380-feae-40e5-9d92-587de358de7d None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:46.532436 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a282f05-f904-4f9f-9e21-f5fe95a14eae None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:46.564478 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:46.572892 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c1888ef-8164-497e-a2ce-ff90386a16f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:46.573755 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c1888ef-8164-497e-a2ce-ff90386a16f4 None admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:46.574287 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c1888ef-8164-497e-a2ce-ff90386a16f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:46.574754 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c1888ef-8164-497e-a2ce-ff90386a16f4 None admin] PATH_INFO: `/v3/users` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:46.575662 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c1888ef-8164-497e-a2ce-ff90386a16f4 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:46.577318 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c1888ef-8164-497e-a2ce-ff90386a16f4 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:46.603516 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7e326c9-d534-4bab-8764-a4fbdd3723d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:46.604881 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e326c9-d534-4bab-8764-a4fbdd3723d0 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:46.605461 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e326c9-d534-4bab-8764-a4fbdd3723d0 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:46.605925 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e326c9-d534-4bab-8764-a4fbdd3723d0 None admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:46.606844 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7e326c9-d534-4bab-8764-a4fbdd3723d0 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:58:46.607323 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7e326c9-d534-4bab-8764-a4fbdd3723d0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:46.608984 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7e326c9-d534-4bab-8764-a4fbdd3723d0 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:46.617035 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 73/144] 10.222.0.77 () {60 vars in 1132 bytes} [Tue Aug 30 04:58:46 2022] POST /identity/v3/users => generated 283 bytes in 271 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 04:58:46.627061 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 72/145] 10.222.0.77 () {56 vars in 1101 bytes} [Tue Aug 30 04:58:46 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 276 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:58:46.635760 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c1888ef-8164-497e-a2ce-ff90386a16f4 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:46.767257 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c1888ef-8164-497e-a2ce-ff90386a16f4 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:46.812719 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7075ffbc-03e8-46c7-8ab8-88987f543c52 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:46.813694 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7075ffbc-03e8-46c7-8ab8-88987f543c52 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:46.814148 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7075ffbc-03e8-46c7-8ab8-88987f543c52 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:46.814689 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7075ffbc-03e8-46c7-8ab8-88987f543c52 None admin] PATH_INFO: `/v3/users/87dcae03963b47e68a40b6afa7553b20` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:46.816056 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7075ffbc-03e8-46c7-8ab8-88987f543c52 None admin] RBAC: Authorizing `identity:get_user(user_id=87dcae03963b47e68a40b6afa7553b20)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:46.817726 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7075ffbc-03e8-46c7-8ab8-88987f543c52 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:46.819669 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 74/146] 10.222.0.77 () {56 vars in 1177 bytes} [Tue Aug 30 04:58:46 2022] GET /identity/v3/users/87dcae03963b47e68a40b6afa7553b20 => generated 289 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:58:46.827970 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7e326c9-d534-4bab-8764-a4fbdd3723d0 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:46.944145 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7e326c9-d534-4bab-8764-a4fbdd3723d0 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:46.980098 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f36fe33d-1d53-478a-953a-211fee484cab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:46.981063 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f36fe33d-1d53-478a-953a-211fee484cab None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:46.981545 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f36fe33d-1d53-478a-953a-211fee484cab None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:46.981984 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f36fe33d-1d53-478a-953a-211fee484cab None admin] PATH_INFO: `/v3/projects/9c3b4147cbea42e3a97c54416188a48c` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:46.983304 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f36fe33d-1d53-478a-953a-211fee484cab None admin] RBAC: Authorizing `identity:get_project(project_id=9c3b4147cbea42e3a97c54416188a48c)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:46.984936 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f36fe33d-1d53-478a-953a-211fee484cab None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:46.986754 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 73/147] 10.222.0.77 () {56 vars in 1186 bytes} [Tue Aug 30 04:58:46 2022] GET /identity/v3/projects/9c3b4147cbea42e3a97c54416188a48c => generated 298 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:58:46.996250 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7075ffbc-03e8-46c7-8ab8-88987f543c52 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:47.065835 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:47.123152 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7075ffbc-03e8-46c7-8ab8-88987f543c52 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:47.167636 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea7a3ec3-c7a6-40f2-9e7b-efdaf5a5f765 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:47.168637 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea7a3ec3-c7a6-40f2-9e7b-efdaf5a5f765 None admin] REQUEST_METHOD: `PUT` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:47.169119 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea7a3ec3-c7a6-40f2-9e7b-efdaf5a5f765 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:47.169565 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea7a3ec3-c7a6-40f2-9e7b-efdaf5a5f765 None admin] PATH_INFO: `/v3/projects/9c3b4147cbea42e3a97c54416188a48c/users/87dcae03963b47e68a40b6afa7553b20/roles/ddaf2963f700481eb93adea7a36f4e73` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:47.173314 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea7a3ec3-c7a6-40f2-9e7b-efdaf5a5f765 None admin] RBAC: Authorizing `identity:create_grant(project_id=9c3b4147cbea42e3a97c54416188a48c, user_id=87dcae03963b47e68a40b6afa7553b20, role_id=ddaf2963f700481eb93adea7a36f4e73)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:47.174982 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea7a3ec3-c7a6-40f2-9e7b-efdaf5a5f765 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:47.203806 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 75/148] 10.222.0.77 () {58 vars in 1439 bytes} [Tue Aug 30 04:58:46 2022] PUT /identity/v3/projects/9c3b4147cbea42e3a97c54416188a48c/users/87dcae03963b47e68a40b6afa7553b20/roles/ddaf2963f700481eb93adea7a36f4e73 => generated 0 bytes in 208 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:58:47.466515 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f89bba3c-ba37-435f-bab2-8dade255f454 None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:47.467492 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f89bba3c-ba37-435f-bab2-8dade255f454 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:47.468488 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f89bba3c-ba37-435f-bab2-8dade255f454 None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:47.470315 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 74/149] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:58:47 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:58:47.475342 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4bce168-5591-4946-90f7-97a28b9fdfdc None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:47.475628 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4bce168-5591-4946-90f7-97a28b9fdfdc None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:47.475901 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4bce168-5591-4946-90f7-97a28b9fdfdc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:47.499128 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-c4bce168-5591-4946-90f7-97a28b9fdfdc None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:58:47.500308 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-c4bce168-5591-4946-90f7-97a28b9fdfdc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:58:47.578257 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 76/150] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:58:47 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:58:47.831784 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31ed5b0e-cc5c-495a-9449-cfeee8e01fce None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:47.832442 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31ed5b0e-cc5c-495a-9449-cfeee8e01fce None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:47.832917 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31ed5b0e-cc5c-495a-9449-cfeee8e01fce None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:47.834455 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 75/151] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:58:47 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:58:47.840303 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99adadd3-f4b8-4da8-9f6b-0827d39c8601 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:47.840587 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99adadd3-f4b8-4da8-9f6b-0827d39c8601 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:47.840861 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99adadd3-f4b8-4da8-9f6b-0827d39c8601 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:47.864059 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-99adadd3-f4b8-4da8-9f6b-0827d39c8601 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:58:47.865251 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-99adadd3-f4b8-4da8-9f6b-0827d39c8601 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:58:47.916482 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:47.953026 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 77/152] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:58:47 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:58:48.013596 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31ed5b0e-cc5c-495a-9449-cfeee8e01fce None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:48.016582 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:48.073633 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4937bd7e-8e2b-424e-934f-a6143fd0687d None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:48.073942 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4937bd7e-8e2b-424e-934f-a6143fd0687d None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:48.074182 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4937bd7e-8e2b-424e-934f-a6143fd0687d None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:48.075554 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 78/153] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:58:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:58:48.081493 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3fb1fee-07db-40d9-9537-20d3ff11f3a3 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:48.081772 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3fb1fee-07db-40d9-9537-20d3ff11f3a3 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:48.082089 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3fb1fee-07db-40d9-9537-20d3ff11f3a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:48.104628 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-c3fb1fee-07db-40d9-9537-20d3ff11f3a3 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:58:48.105811 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-c3fb1fee-07db-40d9-9537-20d3ff11f3a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:58:48.199525 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31ed5b0e-cc5c-495a-9449-cfeee8e01fce None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:48.199969 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 79/154] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:58:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 30 04:58:48.242947 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed4329ae-40f0-4e55-bc7e-3dd618b56ee2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:48.243622 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed4329ae-40f0-4e55-bc7e-3dd618b56ee2 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:48.243902 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed4329ae-40f0-4e55-bc7e-3dd618b56ee2 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:48.244175 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed4329ae-40f0-4e55-bc7e-3dd618b56ee2 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:48.255120 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed4329ae-40f0-4e55-bc7e-3dd618b56ee2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:58:48.261047 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-ed4329ae-40f0-4e55-bc7e-3dd618b56ee2 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 30 04:58:48.261047 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:58:48.261047 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:58:48.261047 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:58:48.261047 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:58:48.261047 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:58:48.261047 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:58:48.261047 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:58:48.261047 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:58:48.261047 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:58:48.261047 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:58:48.261047 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:58:48.261047 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:58:48.261047 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:58:48.261047 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:58:48.261047 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:58:48.261907 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:58:48.261907 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:58:48.261907 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:58:48.261907 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:58:48.261907 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:58:48.261907 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:58:48.261907 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:58:48.261907 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:58:48.261907 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:58:48.261907 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:58:48.261907 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:58:48.261907 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 04:58:48.261907 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:58:48.261907 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:58:48.261907 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:58:48.261907 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:58:48.262653 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:58:48.262653 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:58:48.262653 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:58:48.262653 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:58:48.262653 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:58:48.262653 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:58:48.262653 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:58:48.262653 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:58:48.262653 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:58:48.262653 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:58:48.262653 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 30 04:58:48.262653 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 04:58:48.262653 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 76/155] 10.222.0.77 () {56 vars in 1099 bytes} [Tue Aug 30 04:58:48 2022] GET /identity/v3/roles/member => generated 84 bytes in 249 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:58:48.267950 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3fb1fee-07db-40d9-9537-20d3ff11f3a3 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:48.377146 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3fb1fee-07db-40d9-9537-20d3ff11f3a3 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:48.379477 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:48.416512 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0622c7d3-01f2-449c-a2c8-adf3b7da009d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:48.417139 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0622c7d3-01f2-449c-a2c8-adf3b7da009d None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:48.417419 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0622c7d3-01f2-449c-a2c8-adf3b7da009d None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:48.417686 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0622c7d3-01f2-449c-a2c8-adf3b7da009d None admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:48.418357 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0622c7d3-01f2-449c-a2c8-adf3b7da009d None admin] RBAC: Adding query filter params (name=member) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:58:48.418657 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0622c7d3-01f2-449c-a2c8-adf3b7da009d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:48.420077 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0622c7d3-01f2-449c-a2c8-adf3b7da009d None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:48.435285 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 80/156] 10.222.0.77 () {56 vars in 1101 bytes} [Tue Aug 30 04:58:48 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:58:48.440742 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed4329ae-40f0-4e55-bc7e-3dd618b56ee2 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:48.521425 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:48.564101 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed4329ae-40f0-4e55-bc7e-3dd618b56ee2 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:48.593330 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0622c7d3-01f2-449c-a2c8-adf3b7da009d None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:48.623094 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-986466a1-4375-43af-b34c-69ed036c5d0b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:48.623732 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-986466a1-4375-43af-b34c-69ed036c5d0b None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:48.627398 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-986466a1-4375-43af-b34c-69ed036c5d0b None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:48.627630 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-986466a1-4375-43af-b34c-69ed036c5d0b None admin] PATH_INFO: `/v3/users/bb6b84c639b14608813a9a5e0dac6840` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:48.628854 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-986466a1-4375-43af-b34c-69ed036c5d0b None admin] RBAC: Authorizing `identity:get_user(user_id=bb6b84c639b14608813a9a5e0dac6840)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:48.630329 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-986466a1-4375-43af-b34c-69ed036c5d0b None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:48.631962 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 77/157] 10.222.0.77 () {56 vars in 1177 bytes} [Tue Aug 30 04:58:48 2022] GET /identity/v3/users/bb6b84c639b14608813a9a5e0dac6840 => generated 275 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:58:48.638150 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-986466a1-4375-43af-b34c-69ed036c5d0b None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:48.746484 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0622c7d3-01f2-449c-a2c8-adf3b7da009d None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:48.810963 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-986466a1-4375-43af-b34c-69ed036c5d0b None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:48.812046 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eefb406f-ff77-4194-8078-0a2236503dba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:48.812684 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eefb406f-ff77-4194-8078-0a2236503dba None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:48.813005 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eefb406f-ff77-4194-8078-0a2236503dba None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:48.813229 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eefb406f-ff77-4194-8078-0a2236503dba None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:48.836079 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eefb406f-ff77-4194-8078-0a2236503dba None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:58:48.842048 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-eefb406f-ff77-4194-8078-0a2236503dba None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 30 04:58:48.842048 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:58:48.842048 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:58:48.842048 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:58:48.842048 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:58:48.842048 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:58:48.842048 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:58:48.842048 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:58:48.842048 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:58:48.842048 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:58:48.842048 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:58:48.842048 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:58:48.842048 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:58:48.842048 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:58:48.842048 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:58:48.842048 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:58:48.843116 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:58:48.843116 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:58:48.843116 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:58:48.843116 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:58:48.843116 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:58:48.843116 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:58:48.843116 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:58:48.843116 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:58:48.843116 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:58:48.843116 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:58:48.843116 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:58:48.843116 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 04:58:48.843116 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:58:48.843116 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:58:48.843116 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:58:48.843116 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:58:48.844005 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:58:48.844005 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:58:48.844005 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:58:48.844005 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:58:48.844005 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:58:48.844005 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:58:48.844005 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:58:48.844005 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:58:48.844005 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:58:48.844005 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:58:48.844005 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 30 04:58:48.844005 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 04:58:48.851416 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 81/158] 10.222.0.77 () {56 vars in 1114 bytes} [Tue Aug 30 04:58:48 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 258 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:58:48.852154 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eefb406f-ff77-4194-8078-0a2236503dba None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:48.868598 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:48.878919 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e449b334-bfda-4c3b-bbfc-8ef0014c06e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:48.879482 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e449b334-bfda-4c3b-bbfc-8ef0014c06e6 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:48.879757 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e449b334-bfda-4c3b-bbfc-8ef0014c06e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:48.880023 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e449b334-bfda-4c3b-bbfc-8ef0014c06e6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:48.897268 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:49.019493 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eefb406f-ff77-4194-8078-0a2236503dba None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:49.056465 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e449b334-bfda-4c3b-bbfc-8ef0014c06e6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:49.057933 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e449b334-bfda-4c3b-bbfc-8ef0014c06e6 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:49.096890 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eb4ae72-4319-4e84-8b79-5c59d46466de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:49.097527 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb4ae72-4319-4e84-8b79-5c59d46466de None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:49.097801 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb4ae72-4319-4e84-8b79-5c59d46466de None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:49.098071 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb4ae72-4319-4e84-8b79-5c59d46466de None admin] PATH_INFO: `/v3/roles/member` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:49.121874 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eb4ae72-4319-4e84-8b79-5c59d46466de None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:58:49.129392 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-7eb4ae72-4319-4e84-8b79-5c59d46466de None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 30 04:58:49.129392 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:58:49.129392 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:58:49.129392 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:58:49.129392 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:58:49.129392 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:58:49.129392 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:58:49.129392 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:58:49.129392 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:58:49.129392 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:58:49.129392 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:58:49.129392 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:58:49.129392 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:58:49.129392 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:58:49.129392 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:58:49.129392 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:58:49.130224 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:58:49.130224 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:58:49.130224 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:58:49.130224 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:58:49.130224 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:58:49.130224 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:58:49.130224 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:58:49.130224 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:58:49.130224 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:58:49.130224 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:58:49.130224 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:58:49.130224 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 04:58:49.130224 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:58:49.130224 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:58:49.130224 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:58:49.130224 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:58:49.131094 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:58:49.131094 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:58:49.131094 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:58:49.131094 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:58:49.131094 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:58:49.131094 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:58:49.131094 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:58:49.131094 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:58:49.131094 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:58:49.131094 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:58:49.131094 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 30 04:58:49.131094 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 04:58:49.131094 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 82/159] 10.222.0.77 () {56 vars in 1099 bytes} [Tue Aug 30 04:58:48 2022] GET /identity/v3/roles/member => generated 84 bytes in 277 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:58:49.131094 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36692a9a-e490-4843-9cfe-f9ba3a11e1b9 None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:49.131819 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36692a9a-e490-4843-9cfe-f9ba3a11e1b9 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:49.131819 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36692a9a-e490-4843-9cfe-f9ba3a11e1b9 None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:49.131819 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 83/160] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:58:49 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:58:49.137036 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36692a9a-e490-4843-9cfe-f9ba3a11e1b9 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:49.247593 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 78/161] 10.222.0.77 () {58 vars in 1282 bytes} [Tue Aug 30 04:58:48 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 610 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Aug 30 04:58:49.249946 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e449b334-bfda-4c3b-bbfc-8ef0014c06e6 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:49.317962 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:49.339993 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:49.378610 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36692a9a-e490-4843-9cfe-f9ba3a11e1b9 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:49.444901 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0f7b2d3-4aa8-497f-b29a-c144dee402fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:49.445429 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f7b2d3-4aa8-497f-b29a-c144dee402fe None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:49.445717 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f7b2d3-4aa8-497f-b29a-c144dee402fe None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:49.445990 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f7b2d3-4aa8-497f-b29a-c144dee402fe None admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:49.447439 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0f7b2d3-4aa8-497f-b29a-c144dee402fe None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:58:49.447740 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0f7b2d3-4aa8-497f-b29a-c144dee402fe None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:49.449294 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0f7b2d3-4aa8-497f-b29a-c144dee402fe None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:49.468844 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 84/162] 10.222.0.77 () {56 vars in 1111 bytes} [Tue Aug 30 04:58:49 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 337 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:58:49.470392 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2391cf8f-5bf1-4436-a620-88e79cb98eea None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:49.470679 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2391cf8f-5bf1-4436-a620-88e79cb98eea None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:49.470977 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2391cf8f-5bf1-4436-a620-88e79cb98eea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:49.502252 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-2391cf8f-5bf1-4436-a620-88e79cb98eea None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:58:49.503641 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-2391cf8f-5bf1-4436-a620-88e79cb98eea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:58:49.520967 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e449b334-bfda-4c3b-bbfc-8ef0014c06e6 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:49.598624 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf7a1875-45ef-42ee-9b51-6bbdb2c880b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:49.599311 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf7a1875-45ef-42ee-9b51-6bbdb2c880b3 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:49.599556 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf7a1875-45ef-42ee-9b51-6bbdb2c880b3 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:49.599779 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf7a1875-45ef-42ee-9b51-6bbdb2c880b3 None admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:49.600435 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf7a1875-45ef-42ee-9b51-6bbdb2c880b3 None admin] RBAC: Adding query filter params (name=member) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:58:49.600730 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf7a1875-45ef-42ee-9b51-6bbdb2c880b3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:49.602093 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf7a1875-45ef-42ee-9b51-6bbdb2c880b3 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:49.617170 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 79/163] 10.222.0.77 () {56 vars in 1101 bytes} [Tue Aug 30 04:58:49 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 369 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:58:49.617983 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf7a1875-45ef-42ee-9b51-6bbdb2c880b3 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:49.644100 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 85/164] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:58:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:58:49.645046 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2391cf8f-5bf1-4436-a620-88e79cb98eea None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:49.821099 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:49.843834 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:49.864912 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf7a1875-45ef-42ee-9b51-6bbdb2c880b3 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:49.865174 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2391cf8f-5bf1-4436-a620-88e79cb98eea None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:49.933002 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1fe2e6c-a41c-4ff6-8f80-b8abc1f6bdbc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:49.933640 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1fe2e6c-a41c-4ff6-8f80-b8abc1f6bdbc None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:49.933912 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1fe2e6c-a41c-4ff6-8f80-b8abc1f6bdbc None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:49.934179 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1fe2e6c-a41c-4ff6-8f80-b8abc1f6bdbc None admin] PATH_INFO: `/v3/projects/8b38c5abf3d344cc8144125aa0a9abb1` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:49.939517 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1fe2e6c-a41c-4ff6-8f80-b8abc1f6bdbc None admin] RBAC: Authorizing `identity:get_project(project_id=8b38c5abf3d344cc8144125aa0a9abb1)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:49.940981 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1fe2e6c-a41c-4ff6-8f80-b8abc1f6bdbc None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:49.941316 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c88de7ae-64df-422d-992f-ef216f848192 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:49.941940 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c88de7ae-64df-422d-992f-ef216f848192 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:49.942216 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c88de7ae-64df-422d-992f-ef216f848192 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:49.942485 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c88de7ae-64df-422d-992f-ef216f848192 None admin] PATH_INFO: `/v3/users/bb6b84c639b14608813a9a5e0dac6840` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:49.942652 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 80/165] 10.222.0.77 () {56 vars in 1186 bytes} [Tue Aug 30 04:58:49 2022] GET /identity/v3/projects/8b38c5abf3d344cc8144125aa0a9abb1 => generated 312 bytes in 325 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:58:49.943496 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1fe2e6c-a41c-4ff6-8f80-b8abc1f6bdbc None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:49.943664 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c88de7ae-64df-422d-992f-ef216f848192 None admin] RBAC: Authorizing `identity:get_user(user_id=bb6b84c639b14608813a9a5e0dac6840)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:49.945169 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c88de7ae-64df-422d-992f-ef216f848192 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:49.947341 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 86/166] 10.222.0.77 () {56 vars in 1177 bytes} [Tue Aug 30 04:58:49 2022] GET /identity/v3/users/bb6b84c639b14608813a9a5e0dac6840 => generated 275 bytes in 303 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:58:49.948979 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c88de7ae-64df-422d-992f-ef216f848192 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:50.182251 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c88de7ae-64df-422d-992f-ef216f848192 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:50.204927 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1fe2e6c-a41c-4ff6-8f80-b8abc1f6bdbc None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:50.245401 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e7e912c-f9df-476c-8ab3-2f5a098a33d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:50.246004 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e7e912c-f9df-476c-8ab3-2f5a098a33d0 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:50.246277 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e7e912c-f9df-476c-8ab3-2f5a098a33d0 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:50.246544 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e7e912c-f9df-476c-8ab3-2f5a098a33d0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:50.247292 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e7e912c-f9df-476c-8ab3-2f5a098a33d0 None admin] RBAC: Adding query filter params (role.id=6f03d86980c34c36b44a8d6764ff4fda, scope.project.id=8b38c5abf3d344cc8144125aa0a9abb1, user.id=bb6b84c639b14608813a9a5e0dac6840) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:58:50.247592 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e7e912c-f9df-476c-8ab3-2f5a098a33d0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:50.249037 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e7e912c-f9df-476c-8ab3-2f5a098a33d0 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:50.258127 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 87/167] 10.222.0.77 () {56 vars in 1374 bytes} [Tue Aug 30 04:58:49 2022] GET /identity/v3/role_assignments?user.id=bb6b84c639b14608813a9a5e0dac6840&scope.project.id=8b38c5abf3d344cc8144125aa0a9abb1&role.id=6f03d86980c34c36b44a8d6764ff4fda => generated 608 bytes in 309 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:58:50.258986 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e7e912c-f9df-476c-8ab3-2f5a098a33d0 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:50.273234 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8c296ba-41f0-46af-b2ea-994586b04327 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:50.273885 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c296ba-41f0-46af-b2ea-994586b04327 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:50.274159 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c296ba-41f0-46af-b2ea-994586b04327 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:50.274458 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c296ba-41f0-46af-b2ea-994586b04327 None admin] PATH_INFO: `/v3/users/bb6b84c639b14608813a9a5e0dac6840` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:50.275720 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8c296ba-41f0-46af-b2ea-994586b04327 None admin] RBAC: Authorizing `identity:get_user(user_id=bb6b84c639b14608813a9a5e0dac6840)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:50.277162 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8c296ba-41f0-46af-b2ea-994586b04327 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:50.278922 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 81/168] 10.222.0.77 () {56 vars in 1177 bytes} [Tue Aug 30 04:58:49 2022] GET /identity/v3/users/bb6b84c639b14608813a9a5e0dac6840 => generated 275 bytes in 336 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:58:50.283686 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8c296ba-41f0-46af-b2ea-994586b04327 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:50.288831 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:50.328436 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:50.467724 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e7e912c-f9df-476c-8ab3-2f5a098a33d0 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:50.537291 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86380cbd-dda4-45b4-bc42-c3973b547213 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:50.537909 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86380cbd-dda4-45b4-bc42-c3973b547213 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:50.538189 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86380cbd-dda4-45b4-bc42-c3973b547213 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:50.538452 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86380cbd-dda4-45b4-bc42-c3973b547213 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:50.544918 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8c296ba-41f0-46af-b2ea-994586b04327 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:50.612234 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90e0f99c-67bc-44f3-8866-e64b5f147244 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:50.612870 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90e0f99c-67bc-44f3-8866-e64b5f147244 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:50.613122 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90e0f99c-67bc-44f3-8866-e64b5f147244 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:50.613388 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90e0f99c-67bc-44f3-8866-e64b5f147244 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:50.640762 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90e0f99c-67bc-44f3-8866-e64b5f147244 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:58:50.649387 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-90e0f99c-67bc-44f3-8866-e64b5f147244 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 04:58:50.649387 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:58:50.649387 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:58:50.649387 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:58:50.649387 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:58:50.649387 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:58:50.649387 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:58:50.649387 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:58:50.649387 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:58:50.649387 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:58:50.649387 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:58:50.649387 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:58:50.649387 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:58:50.649387 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:58:50.649387 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:58:50.649387 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:58:50.650262 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:58:50.650262 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:58:50.650262 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:58:50.650262 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:58:50.650262 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:58:50.650262 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:58:50.650262 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:58:50.650262 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:58:50.650262 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:58:50.650262 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:58:50.650262 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:58:50.650262 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 04:58:50.650262 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:58:50.650262 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:58:50.650262 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:58:50.650262 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:58:50.651064 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:58:50.651064 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:58:50.651064 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:58:50.651064 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:58:50.651064 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:58:50.651064 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:58:50.651064 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:58:50.651064 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:58:50.651064 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:58:50.651064 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:58:50.651064 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 04:58:50.651064 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 04:58:50.651064 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 82/169] 10.222.0.77 () {56 vars in 1096 bytes} [Tue Aug 30 04:58:50 2022] GET /identity/v3/roles/admin => generated 83 bytes in 371 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:58:50.651064 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90e0f99c-67bc-44f3-8866-e64b5f147244 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:50.671602 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86380cbd-dda4-45b4-bc42-c3973b547213 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:50.673225 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86380cbd-dda4-45b4-bc42-c3973b547213 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:50.685966 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:50.784317 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90e0f99c-67bc-44f3-8866-e64b5f147244 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:50.787154 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:50.803961 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 88/170] 10.222.0.77 () {58 vars in 1282 bytes} [Tue Aug 30 04:58:50 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 545 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 04:58:50.806415 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096b58ae-639f-4407-9249-87fae69a9ade None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:50.806696 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096b58ae-639f-4407-9249-87fae69a9ade None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:50.809471 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096b58ae-639f-4407-9249-87fae69a9ade None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:50.810639 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 89/171] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:58:50 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:58:50.815926 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76e1f402-d394-4b0e-bdbc-490a6783af53 None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:50.816201 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76e1f402-d394-4b0e-bdbc-490a6783af53 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:50.816469 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76e1f402-d394-4b0e-bdbc-490a6783af53 None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:50.818019 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 90/172] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:58:50 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:58:50.819205 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76e1f402-d394-4b0e-bdbc-490a6783af53 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:50.846465 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f24727f4-ba8b-40bf-876b-5b1bd254d5fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:50.847692 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f24727f4-ba8b-40bf-876b-5b1bd254d5fb None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:50.848154 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f24727f4-ba8b-40bf-876b-5b1bd254d5fb None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:50.848622 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f24727f4-ba8b-40bf-876b-5b1bd254d5fb None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:50.941377 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f24727f4-ba8b-40bf-876b-5b1bd254d5fb None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:50.943904 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f24727f4-ba8b-40bf-876b-5b1bd254d5fb None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:50.952906 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76e1f402-d394-4b0e-bdbc-490a6783af53 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:50.965653 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:50.992363 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0615c8e1-a52a-4c51-8147-3421e9492cb2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:50.993119 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0615c8e1-a52a-4c51-8147-3421e9492cb2 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:50.993520 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0615c8e1-a52a-4c51-8147-3421e9492cb2 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:50.993932 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0615c8e1-a52a-4c51-8147-3421e9492cb2 None admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:50.994732 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0615c8e1-a52a-4c51-8147-3421e9492cb2 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:58:50.995150 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0615c8e1-a52a-4c51-8147-3421e9492cb2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:50.996679 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0615c8e1-a52a-4c51-8147-3421e9492cb2 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:51.006200 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 91/173] 10.222.0.77 () {56 vars in 1099 bytes} [Tue Aug 30 04:58:50 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:58:51.007186 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0615c8e1-a52a-4c51-8147-3421e9492cb2 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:51.063882 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:51.082334 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 83/174] 10.222.0.77 () {58 vars in 1282 bytes} [Tue Aug 30 04:58:50 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 432 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 04:58:51.083946 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a44f14b-6515-4aac-9a43-18729d36ff6c None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:51.084450 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a44f14b-6515-4aac-9a43-18729d36ff6c None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:51.084886 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a44f14b-6515-4aac-9a43-18729d36ff6c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:51.105238 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-4a44f14b-6515-4aac-9a43-18729d36ff6c None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:58:51.106479 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-4a44f14b-6515-4aac-9a43-18729d36ff6c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:58:51.121018 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0615c8e1-a52a-4c51-8147-3421e9492cb2 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:51.158763 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1809675-4645-4692-9078-22418d17f2e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:51.159687 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1809675-4645-4692-9078-22418d17f2e9 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:51.160148 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1809675-4645-4692-9078-22418d17f2e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:51.160679 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1809675-4645-4692-9078-22418d17f2e9 None admin] PATH_INFO: `/v3/projects/9c3b4147cbea42e3a97c54416188a48c` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:51.161982 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1809675-4645-4692-9078-22418d17f2e9 None admin] RBAC: Authorizing `identity:get_project(project_id=9c3b4147cbea42e3a97c54416188a48c)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:51.163840 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1809675-4645-4692-9078-22418d17f2e9 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:51.170554 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 92/175] 10.222.0.77 () {56 vars in 1186 bytes} [Tue Aug 30 04:58:51 2022] GET /identity/v3/projects/9c3b4147cbea42e3a97c54416188a48c => generated 298 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:58:51.171945 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecfd8126-5ea5-44e7-853b-b2fec56ce26e None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:51.172416 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecfd8126-5ea5-44e7-853b-b2fec56ce26e None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:51.172764 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecfd8126-5ea5-44e7-853b-b2fec56ce26e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:51.173915 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 84/176] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:58:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:58:51.180740 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a44f14b-6515-4aac-9a43-18729d36ff6c None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:51.190787 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-ecfd8126-5ea5-44e7-853b-b2fec56ce26e None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:58:51.192017 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-ecfd8126-5ea5-44e7-853b-b2fec56ce26e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:58:51.238372 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:51.285092 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 93/177] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:58:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:58:51.286645 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd74ed6e-d0cb-451e-a02f-ae4fc075d295 None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:51.287143 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd74ed6e-d0cb-451e-a02f-ae4fc075d295 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:51.287625 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd74ed6e-d0cb-451e-a02f-ae4fc075d295 None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:51.289153 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 94/178] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:58:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:58:51.290044 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd74ed6e-d0cb-451e-a02f-ae4fc075d295 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:51.316400 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a44f14b-6515-4aac-9a43-18729d36ff6c None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:51.359220 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:51.364437 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-018f2120-abc7-4065-baf8-b1b7aab1e793 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:51.365064 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-018f2120-abc7-4065-baf8-b1b7aab1e793 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:51.365342 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-018f2120-abc7-4065-baf8-b1b7aab1e793 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:51.365741 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-018f2120-abc7-4065-baf8-b1b7aab1e793 None admin] PATH_INFO: `/v3/users/fdab92bac3f64739a0fcb5f729d40f85` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:51.367141 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-018f2120-abc7-4065-baf8-b1b7aab1e793 None admin] RBAC: Authorizing `identity:get_user(user_id=fdab92bac3f64739a0fcb5f729d40f85)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:51.369455 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-018f2120-abc7-4065-baf8-b1b7aab1e793 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:51.371503 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 85/179] 10.222.0.77 () {56 vars in 1177 bytes} [Tue Aug 30 04:58:51 2022] GET /identity/v3/users/fdab92bac3f64739a0fcb5f729d40f85 => generated 247 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:58:51.372077 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-018f2120-abc7-4065-baf8-b1b7aab1e793 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:51.419033 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd74ed6e-d0cb-451e-a02f-ae4fc075d295 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:51.464177 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71b80dcb-7b98-4b0b-8eeb-702e69425ec8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:51.465724 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b80dcb-7b98-4b0b-8eeb-702e69425ec8 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:51.466006 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b80dcb-7b98-4b0b-8eeb-702e69425ec8 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:51.466284 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b80dcb-7b98-4b0b-8eeb-702e69425ec8 None admin] PATH_INFO: `/v3/projects/9c3b4147cbea42e3a97c54416188a48c` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:51.467422 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71b80dcb-7b98-4b0b-8eeb-702e69425ec8 None admin] RBAC: Authorizing `identity:get_project(project_id=9c3b4147cbea42e3a97c54416188a48c)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:51.468866 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71b80dcb-7b98-4b0b-8eeb-702e69425ec8 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:51.470679 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 95/180] 10.222.0.77 () {56 vars in 1186 bytes} [Tue Aug 30 04:58:51 2022] GET /identity/v3/projects/9c3b4147cbea42e3a97c54416188a48c => generated 298 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:58:51.471648 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb211e53-b993-4929-994f-b96723f04a9e None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:51.471925 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb211e53-b993-4929-994f-b96723f04a9e None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:51.472189 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb211e53-b993-4929-994f-b96723f04a9e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:51.482646 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-018f2120-abc7-4065-baf8-b1b7aab1e793 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:51.490522 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-fb211e53-b993-4929-994f-b96723f04a9e None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:58:51.491893 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-fb211e53-b993-4929-994f-b96723f04a9e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:58:51.518157 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb4dcc71-cc7f-43f3-8dd4-76e056ef93fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:51.518986 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb4dcc71-cc7f-43f3-8dd4-76e056ef93fd None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:51.519265 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb4dcc71-cc7f-43f3-8dd4-76e056ef93fd None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:51.519536 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb4dcc71-cc7f-43f3-8dd4-76e056ef93fd None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:51.520256 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb4dcc71-cc7f-43f3-8dd4-76e056ef93fd None admin] RBAC: Adding query filter params (role.id=3b9d5509a2f7478c9b4989d921daef2a, scope.project.id=9c3b4147cbea42e3a97c54416188a48c, user.id=bb6b84c639b14608813a9a5e0dac6840) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:58:51.520563 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb4dcc71-cc7f-43f3-8dd4-76e056ef93fd None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:51.522090 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb4dcc71-cc7f-43f3-8dd4-76e056ef93fd None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:51.524715 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:51.529425 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 86/181] 10.222.0.77 () {56 vars in 1374 bytes} [Tue Aug 30 04:58:51 2022] GET /identity/v3/role_assignments?user.id=bb6b84c639b14608813a9a5e0dac6840&scope.project.id=9c3b4147cbea42e3a97c54416188a48c&role.id=3b9d5509a2f7478c9b4989d921daef2a => generated 608 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:58:51.530555 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-551d8c21-5877-47f7-9951-a10e07ed96fa None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:51.530947 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-551d8c21-5877-47f7-9951-a10e07ed96fa None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:51.531146 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-551d8c21-5877-47f7-9951-a10e07ed96fa None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:51.532354 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 87/182] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:58:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:58:51.533039 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-551d8c21-5877-47f7-9951-a10e07ed96fa None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:51.588905 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 96/183] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:58:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:58:51.589767 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb211e53-b993-4929-994f-b96723f04a9e None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:51.618565 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:51.686031 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-551d8c21-5877-47f7-9951-a10e07ed96fa None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:51.707894 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb211e53-b993-4929-994f-b96723f04a9e None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:51.726831 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15d433a6-1e6a-4b1c-8291-84e10d5fcb38 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:51.728175 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15d433a6-1e6a-4b1c-8291-84e10d5fcb38 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:51.728453 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15d433a6-1e6a-4b1c-8291-84e10d5fcb38 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:51.729018 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15d433a6-1e6a-4b1c-8291-84e10d5fcb38 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:51.749135 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9f63c48-ff27-4207-b097-137b68ca2466 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:51.749773 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f63c48-ff27-4207-b097-137b68ca2466 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:51.750052 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f63c48-ff27-4207-b097-137b68ca2466 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:51.750321 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f63c48-ff27-4207-b097-137b68ca2466 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:51.761781 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9f63c48-ff27-4207-b097-137b68ca2466 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:58:51.766677 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-f9f63c48-ff27-4207-b097-137b68ca2466 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 30 04:58:51.766677 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:58:51.766677 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:58:51.766677 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:58:51.766677 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:58:51.766677 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:58:51.766677 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:58:51.766677 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:58:51.766677 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:58:51.766677 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:58:51.766677 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:58:51.766677 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:58:51.766677 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:58:51.766677 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:58:51.766677 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:58:51.766677 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:58:51.767789 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:58:51.767789 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:58:51.767789 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:58:51.767789 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:58:51.767789 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:58:51.767789 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:58:51.767789 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:58:51.767789 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:58:51.767789 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:58:51.767789 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:58:51.767789 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:58:51.767789 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 04:58:51.767789 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:58:51.767789 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:58:51.767789 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:58:51.767789 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:58:51.768585 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:58:51.768585 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:58:51.768585 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:58:51.768585 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:58:51.768585 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:58:51.768585 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:58:51.768585 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:58:51.768585 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:58:51.768585 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:58:51.768585 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:58:51.768585 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 30 04:58:51.768585 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 04:58:51.769252 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 97/184] 10.222.0.77 () {56 vars in 1099 bytes} [Tue Aug 30 04:58:51 2022] GET /identity/v3/roles/reader => generated 84 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:58:51.769621 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9f63c48-ff27-4207-b097-137b68ca2466 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:51.799757 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:51.819606 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15d433a6-1e6a-4b1c-8291-84e10d5fcb38 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:51.821169 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15d433a6-1e6a-4b1c-8291-84e10d5fcb38 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:51.875181 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:51.888821 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9f63c48-ff27-4207-b097-137b68ca2466 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:51.922662 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e4a34e1-50d4-4baf-a6c1-a250cea2fc71 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:51.923284 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e4a34e1-50d4-4baf-a6c1-a250cea2fc71 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:51.923556 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e4a34e1-50d4-4baf-a6c1-a250cea2fc71 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:51.923823 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e4a34e1-50d4-4baf-a6c1-a250cea2fc71 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:51.924533 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e4a34e1-50d4-4baf-a6c1-a250cea2fc71 None admin] RBAC: Adding query filter params (role.id=6f03d86980c34c36b44a8d6764ff4fda, scope.project.id=9c3b4147cbea42e3a97c54416188a48c, user.id=bb6b84c639b14608813a9a5e0dac6840) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:58:51.924833 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e4a34e1-50d4-4baf-a6c1-a250cea2fc71 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:51.926261 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e4a34e1-50d4-4baf-a6c1-a250cea2fc71 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:51.927340 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 88/185] 10.222.0.77 () {58 vars in 1282 bytes} [Tue Aug 30 04:58:51 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 394 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 04:58:51.928644 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d204cc-0ee6-49de-99a5-fe8c41829446 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:51.928931 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d204cc-0ee6-49de-99a5-fe8c41829446 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:51.929207 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d204cc-0ee6-49de-99a5-fe8c41829446 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:51.936755 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 98/186] 10.222.0.77 () {56 vars in 1374 bytes} [Tue Aug 30 04:58:51 2022] GET /identity/v3/role_assignments?user.id=bb6b84c639b14608813a9a5e0dac6840&scope.project.id=9c3b4147cbea42e3a97c54416188a48c&role.id=6f03d86980c34c36b44a8d6764ff4fda => generated 608 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:58:51.937535 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e4a34e1-50d4-4baf-a6c1-a250cea2fc71 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:51.948436 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-46d204cc-0ee6-49de-99a5-fe8c41829446 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:58:51.949698 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-46d204cc-0ee6-49de-99a5-fe8c41829446 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:58:52.026908 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 89/187] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:58:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:58:52.035835 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46d204cc-0ee6-49de-99a5-fe8c41829446 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:52.039993 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:52.057853 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e4a34e1-50d4-4baf-a6c1-a250cea2fc71 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:52.104752 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4c8bc0d-189b-40d0-beff-9fc9715e0064 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:52.105389 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c8bc0d-189b-40d0-beff-9fc9715e0064 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:52.105678 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c8bc0d-189b-40d0-beff-9fc9715e0064 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:52.105958 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c8bc0d-189b-40d0-beff-9fc9715e0064 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:52.120938 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4c8bc0d-189b-40d0-beff-9fc9715e0064 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:58:52.127978 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-d4c8bc0d-189b-40d0-beff-9fc9715e0064 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 04:58:52.127978 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:58:52.127978 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:58:52.127978 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:58:52.127978 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:58:52.127978 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:58:52.127978 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:58:52.127978 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:58:52.127978 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:58:52.127978 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:58:52.127978 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:58:52.127978 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:58:52.127978 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:58:52.127978 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:58:52.127978 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:58:52.127978 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:58:52.128840 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:58:52.128840 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:58:52.128840 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:58:52.128840 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:58:52.128840 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:58:52.128840 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:58:52.128840 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:58:52.128840 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:58:52.128840 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:58:52.128840 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:58:52.128840 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:58:52.128840 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 04:58:52.128840 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:58:52.128840 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:58:52.128840 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:58:52.128840 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:58:52.129639 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:58:52.129639 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:58:52.129639 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:58:52.129639 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:58:52.129639 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:58:52.129639 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:58:52.129639 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:58:52.129639 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:58:52.129639 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:58:52.129639 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:58:52.129639 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 04:58:52.129639 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 04:58:52.129639 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 99/188] 10.222.0.77 () {56 vars in 1096 bytes} [Tue Aug 30 04:58:51 2022] GET /identity/v3/roles/admin => generated 83 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:58:52.130291 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4c8bc0d-189b-40d0-beff-9fc9715e0064 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:52.138474 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:52.172092 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46d204cc-0ee6-49de-99a5-fe8c41829446 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:52.216795 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fe8a9d7-eae3-4d68-85eb-d2faf7432939 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:52.217401 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe8a9d7-eae3-4d68-85eb-d2faf7432939 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:52.217679 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe8a9d7-eae3-4d68-85eb-d2faf7432939 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:52.217943 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe8a9d7-eae3-4d68-85eb-d2faf7432939 None admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:52.218592 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fe8a9d7-eae3-4d68-85eb-d2faf7432939 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:58:52.218918 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fe8a9d7-eae3-4d68-85eb-d2faf7432939 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:52.220466 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fe8a9d7-eae3-4d68-85eb-d2faf7432939 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:52.230075 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 90/189] 10.222.0.77 () {56 vars in 1101 bytes} [Tue Aug 30 04:58:52 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:58:52.230666 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fe8a9d7-eae3-4d68-85eb-d2faf7432939 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:52.251074 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4c8bc0d-189b-40d0-beff-9fc9715e0064 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:52.286342 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e029319-fc7d-4f82-8807-c4201b531387 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:52.287004 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e029319-fc7d-4f82-8807-c4201b531387 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:52.287281 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e029319-fc7d-4f82-8807-c4201b531387 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:52.287553 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e029319-fc7d-4f82-8807-c4201b531387 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:52.298327 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e029319-fc7d-4f82-8807-c4201b531387 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:58:52.303485 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-3e029319-fc7d-4f82-8807-c4201b531387 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 30 04:58:52.303485 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:58:52.303485 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:58:52.303485 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:58:52.303485 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:58:52.303485 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:58:52.303485 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:58:52.303485 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:58:52.303485 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:58:52.303485 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:58:52.303485 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:58:52.303485 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:58:52.303485 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:58:52.303485 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:58:52.303485 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:58:52.303485 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:58:52.304395 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:58:52.304395 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:58:52.304395 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:58:52.304395 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:58:52.304395 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:58:52.304395 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:58:52.304395 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:58:52.304395 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:58:52.304395 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:58:52.304395 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:58:52.304395 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:58:52.304395 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 04:58:52.304395 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:58:52.304395 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:58:52.304395 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:58:52.304395 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:58:52.305169 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:58:52.305169 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:58:52.305169 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:58:52.305169 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:58:52.305169 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:58:52.305169 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:58:52.305169 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:58:52.305169 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:58:52.305169 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:58:52.305169 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:58:52.305169 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 30 04:58:52.305169 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 04:58:52.305169 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 100/190] 10.222.0.77 () {56 vars in 1114 bytes} [Tue Aug 30 04:58:52 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:58:52.305859 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e029319-fc7d-4f82-8807-c4201b531387 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:52.320953 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:52.337669 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fe8a9d7-eae3-4d68-85eb-d2faf7432939 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:52.370589 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea14740e-4ce5-4367-8ef2-247dac9530ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:52.371228 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea14740e-4ce5-4367-8ef2-247dac9530ab None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:52.371505 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea14740e-4ce5-4367-8ef2-247dac9530ab None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:52.371774 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea14740e-4ce5-4367-8ef2-247dac9530ab None admin] PATH_INFO: `/v3/projects/9c3b4147cbea42e3a97c54416188a48c` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:52.372872 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea14740e-4ce5-4367-8ef2-247dac9530ab None admin] RBAC: Authorizing `identity:get_project(project_id=9c3b4147cbea42e3a97c54416188a48c)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:52.374292 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea14740e-4ce5-4367-8ef2-247dac9530ab None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:52.375918 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 91/191] 10.222.0.77 () {56 vars in 1186 bytes} [Tue Aug 30 04:58:52 2022] GET /identity/v3/projects/9c3b4147cbea42e3a97c54416188a48c => generated 298 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:58:52.376666 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea14740e-4ce5-4367-8ef2-247dac9530ab None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:52.388540 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:52.410302 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e029319-fc7d-4f82-8807-c4201b531387 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:52.442889 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d170d7e-0d37-4e1b-94c3-4a7c38d2346b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:52.443516 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d170d7e-0d37-4e1b-94c3-4a7c38d2346b None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:52.443794 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d170d7e-0d37-4e1b-94c3-4a7c38d2346b None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:52.444075 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d170d7e-0d37-4e1b-94c3-4a7c38d2346b None admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:52.444736 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d170d7e-0d37-4e1b-94c3-4a7c38d2346b None admin] RBAC: Adding query filter params (name=admin) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:58:52.445039 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d170d7e-0d37-4e1b-94c3-4a7c38d2346b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:52.446945 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d170d7e-0d37-4e1b-94c3-4a7c38d2346b None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:52.454859 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 101/192] 10.222.0.77 () {56 vars in 1099 bytes} [Tue Aug 30 04:58:52 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:58:52.456648 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d170d7e-0d37-4e1b-94c3-4a7c38d2346b None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:52.480976 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea14740e-4ce5-4367-8ef2-247dac9530ab None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:52.513540 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e74cdbc-e526-47fd-9831-53a5076b871c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:52.514152 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e74cdbc-e526-47fd-9831-53a5076b871c None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:52.514426 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e74cdbc-e526-47fd-9831-53a5076b871c None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:52.514700 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e74cdbc-e526-47fd-9831-53a5076b871c None admin] PATH_INFO: `/v3/users/87dcae03963b47e68a40b6afa7553b20` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:52.515837 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e74cdbc-e526-47fd-9831-53a5076b871c None admin] RBAC: Authorizing `identity:get_user(user_id=87dcae03963b47e68a40b6afa7553b20)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:52.517270 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e74cdbc-e526-47fd-9831-53a5076b871c None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:52.518835 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 92/193] 10.222.0.77 () {56 vars in 1177 bytes} [Tue Aug 30 04:58:52 2022] GET /identity/v3/users/87dcae03963b47e68a40b6afa7553b20 => generated 289 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:58:52.519657 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e74cdbc-e526-47fd-9831-53a5076b871c None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:52.536872 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:52.561611 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d170d7e-0d37-4e1b-94c3-4a7c38d2346b None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:52.595001 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f75973f-d027-4af0-b827-ae6a11ecfbae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:52.595618 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f75973f-d027-4af0-b827-ae6a11ecfbae None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:52.595890 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f75973f-d027-4af0-b827-ae6a11ecfbae None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:52.596155 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f75973f-d027-4af0-b827-ae6a11ecfbae None admin] PATH_INFO: `/v3/domains/default` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:52.596936 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:52.597237 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f75973f-d027-4af0-b827-ae6a11ecfbae None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:52.598674 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f75973f-d027-4af0-b827-ae6a11ecfbae None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:52.600257 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 102/194] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 04:58:52 2022] GET /identity/v3/domains/default => generated 199 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:58:52.600257 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f75973f-d027-4af0-b827-ae6a11ecfbae None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:52.621093 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e74cdbc-e526-47fd-9831-53a5076b871c None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:52.655178 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff605c93-4902-40fb-8285-9d849586d99e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:52.655710 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff605c93-4902-40fb-8285-9d849586d99e None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:52.655985 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff605c93-4902-40fb-8285-9d849586d99e None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:52.656258 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff605c93-4902-40fb-8285-9d849586d99e None admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:52.656904 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff605c93-4902-40fb-8285-9d849586d99e None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:58:52.657200 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff605c93-4902-40fb-8285-9d849586d99e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:52.658561 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff605c93-4902-40fb-8285-9d849586d99e None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:52.666364 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 93/195] 10.222.0.77 () {56 vars in 1111 bytes} [Tue Aug 30 04:58:52 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:58:52.667072 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff605c93-4902-40fb-8285-9d849586d99e None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:52.701766 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f75973f-d027-4af0-b827-ae6a11ecfbae None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:52.734607 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70926dfc-133e-4203-aca5-4ab992f3d0dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:52.735242 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70926dfc-133e-4203-aca5-4ab992f3d0dd None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:52.735488 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70926dfc-133e-4203-aca5-4ab992f3d0dd None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:52.735756 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70926dfc-133e-4203-aca5-4ab992f3d0dd None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:52.736463 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70926dfc-133e-4203-aca5-4ab992f3d0dd None admin] RBAC: Adding query filter params (role.id=9d9d5d2c270146488249b7ed943280db, scope.project.id=9c3b4147cbea42e3a97c54416188a48c, user.id=fdab92bac3f64739a0fcb5f729d40f85) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:58:52.736768 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70926dfc-133e-4203-aca5-4ab992f3d0dd None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:52.738196 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70926dfc-133e-4203-aca5-4ab992f3d0dd None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:52.744050 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 103/196] 10.222.0.77 () {56 vars in 1374 bytes} [Tue Aug 30 04:58:52 2022] GET /identity/v3/role_assignments?user.id=fdab92bac3f64739a0fcb5f729d40f85&scope.project.id=9c3b4147cbea42e3a97c54416188a48c&role.id=9d9d5d2c270146488249b7ed943280db => generated 608 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:58:52.744872 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70926dfc-133e-4203-aca5-4ab992f3d0dd None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:52.755287 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:52.767517 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff605c93-4902-40fb-8285-9d849586d99e None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:52.807805 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bd5c003-ffc3-4581-b490-805467577c97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:52.808394 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bd5c003-ffc3-4581-b490-805467577c97 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:52.808735 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bd5c003-ffc3-4581-b490-805467577c97 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:52.809007 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bd5c003-ffc3-4581-b490-805467577c97 None admin] PATH_INFO: `/v3/users/212278f6393840e1848128c611251d64` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:52.822297 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:52.826488 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bd5c003-ffc3-4581-b490-805467577c97 None admin] RBAC: Authorizing `identity:get_user(user_id=212278f6393840e1848128c611251d64)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:52.828037 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bd5c003-ffc3-4581-b490-805467577c97 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:52.834403 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 94/197] 10.222.0.77 () {56 vars in 1177 bytes} [Tue Aug 30 04:58:52 2022] GET /identity/v3/users/212278f6393840e1848128c611251d64 => generated 283 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:58:52.834403 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bd5c003-ffc3-4581-b490-805467577c97 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:52.874795 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70926dfc-133e-4203-aca5-4ab992f3d0dd None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:52.913271 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d11dcc2-7de8-4d81-a4f9-92b4b6567fe3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:52.914176 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d11dcc2-7de8-4d81-a4f9-92b4b6567fe3 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:52.914577 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d11dcc2-7de8-4d81-a4f9-92b4b6567fe3 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:52.914874 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d11dcc2-7de8-4d81-a4f9-92b4b6567fe3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:52.957116 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bd5c003-ffc3-4581-b490-805467577c97 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:52.990751 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fc54bd9-92d3-436d-9f06-dd1d13e27c0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:52.991372 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc54bd9-92d3-436d-9f06-dd1d13e27c0e None admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:52.991646 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc54bd9-92d3-436d-9f06-dd1d13e27c0e None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:52.991912 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc54bd9-92d3-436d-9f06-dd1d13e27c0e None admin] PATH_INFO: `/v3/users` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:52.992547 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fc54bd9-92d3-436d-9f06-dd1d13e27c0e None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:52.994476 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fc54bd9-92d3-436d-9f06-dd1d13e27c0e None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:52.996119 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d11dcc2-7de8-4d81-a4f9-92b4b6567fe3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:52.997577 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d11dcc2-7de8-4d81-a4f9-92b4b6567fe3 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:53.005259 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:53.015800 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 95/198] 10.222.0.77 () {60 vars in 1132 bytes} [Tue Aug 30 04:58:52 2022] POST /identity/v3/users => generated 297 bytes in 185 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:58:53.024097 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fc54bd9-92d3-436d-9f06-dd1d13e27c0e None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:53.087756 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:53.113539 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 104/199] 10.222.0.77 () {58 vars in 1282 bytes} [Tue Aug 30 04:58:52 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 369 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 04:58:53.114763 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d11dcc2-7de8-4d81-a4f9-92b4b6567fe3 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:53.148857 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fc54bd9-92d3-436d-9f06-dd1d13e27c0e None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:53.189894 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1de9e0f9-3312-4ae3-904b-166e4e310500 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:53.190453 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de9e0f9-3312-4ae3-904b-166e4e310500 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:53.190728 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de9e0f9-3312-4ae3-904b-166e4e310500 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:53.192081 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de9e0f9-3312-4ae3-904b-166e4e310500 None admin] PATH_INFO: `/v3/users/212278f6393840e1848128c611251d64` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:53.193736 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1de9e0f9-3312-4ae3-904b-166e4e310500 None admin] RBAC: Authorizing `identity:get_user(user_id=212278f6393840e1848128c611251d64)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:53.195229 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1de9e0f9-3312-4ae3-904b-166e4e310500 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:53.197198 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 96/200] 10.222.0.77 () {56 vars in 1177 bytes} [Tue Aug 30 04:58:53 2022] GET /identity/v3/users/212278f6393840e1848128c611251d64 => generated 283 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:58:53.198128 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1de9e0f9-3312-4ae3-904b-166e4e310500 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:53.235390 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d11dcc2-7de8-4d81-a4f9-92b4b6567fe3 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:53.247877 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:53.281746 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acdce037-cf0a-4fc2-ad98-8f316111efac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:53.282369 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acdce037-cf0a-4fc2-ad98-8f316111efac None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:53.282651 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acdce037-cf0a-4fc2-ad98-8f316111efac None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:53.283048 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acdce037-cf0a-4fc2-ad98-8f316111efac None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:53.328472 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1de9e0f9-3312-4ae3-904b-166e4e310500 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:53.358578 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:53.373934 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c53fe5c-dec5-4998-acf0-5b8474401d17 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:53.374664 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c53fe5c-dec5-4998-acf0-5b8474401d17 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:53.375034 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c53fe5c-dec5-4998-acf0-5b8474401d17 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:53.375377 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c53fe5c-dec5-4998-acf0-5b8474401d17 None admin] PATH_INFO: `/v3/projects/9c3b4147cbea42e3a97c54416188a48c` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:53.376908 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acdce037-cf0a-4fc2-ad98-8f316111efac None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:53.377760 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c53fe5c-dec5-4998-acf0-5b8474401d17 None admin] RBAC: Authorizing `identity:get_project(project_id=9c3b4147cbea42e3a97c54416188a48c)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:53.378776 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acdce037-cf0a-4fc2-ad98-8f316111efac None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:53.379371 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c53fe5c-dec5-4998-acf0-5b8474401d17 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:53.384054 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 97/201] 10.222.0.77 () {56 vars in 1186 bytes} [Tue Aug 30 04:58:53 2022] GET /identity/v3/projects/9c3b4147cbea42e3a97c54416188a48c => generated 298 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:58:53.384054 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c53fe5c-dec5-4998-acf0-5b8474401d17 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:53.497340 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 105/202] 10.222.0.77 () {58 vars in 1282 bytes} [Tue Aug 30 04:58:53 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 383 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 04:58:53.498537 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acdce037-cf0a-4fc2-ad98-8f316111efac None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:53.507256 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:53.526931 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c53fe5c-dec5-4998-acf0-5b8474401d17 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:53.567494 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f34b1ebe-19c5-42a8-b2af-8e677d596ce6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:53.568422 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f34b1ebe-19c5-42a8-b2af-8e677d596ce6 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:53.568530 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f34b1ebe-19c5-42a8-b2af-8e677d596ce6 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:53.568817 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f34b1ebe-19c5-42a8-b2af-8e677d596ce6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:53.608626 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acdce037-cf0a-4fc2-ad98-8f316111efac None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:53.630977 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:53.643756 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f34b1ebe-19c5-42a8-b2af-8e677d596ce6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:53.645544 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f34b1ebe-19c5-42a8-b2af-8e677d596ce6 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:53.646108 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8f3e1b2-8ea8-475c-aa26-9f6f9015ad1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:53.647012 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8f3e1b2-8ea8-475c-aa26-9f6f9015ad1b None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:53.647504 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8f3e1b2-8ea8-475c-aa26-9f6f9015ad1b None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:53.647966 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8f3e1b2-8ea8-475c-aa26-9f6f9015ad1b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:53.648881 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8f3e1b2-8ea8-475c-aa26-9f6f9015ad1b None admin] RBAC: Adding query filter params (role.id=ddaf2963f700481eb93adea7a36f4e73, scope.project.id=9c3b4147cbea42e3a97c54416188a48c, user.id=87dcae03963b47e68a40b6afa7553b20) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:58:53.649923 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8f3e1b2-8ea8-475c-aa26-9f6f9015ad1b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:53.651575 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8f3e1b2-8ea8-475c-aa26-9f6f9015ad1b None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:53.658152 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 106/203] 10.222.0.77 () {56 vars in 1374 bytes} [Tue Aug 30 04:58:53 2022] GET /identity/v3/role_assignments?user.id=87dcae03963b47e68a40b6afa7553b20&scope.project.id=9c3b4147cbea42e3a97c54416188a48c&role.id=ddaf2963f700481eb93adea7a36f4e73 => generated 608 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:58:53.668029 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8f3e1b2-8ea8-475c-aa26-9f6f9015ad1b None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:53.759188 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 98/204] 10.222.0.77 () {58 vars in 1282 bytes} [Tue Aug 30 04:58:53 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 374 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 04:58:53.764930 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f34b1ebe-19c5-42a8-b2af-8e677d596ce6 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:53.821011 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:53.869271 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:53.879927 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f34b1ebe-19c5-42a8-b2af-8e677d596ce6 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:53.920475 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efe23782-c502-4028-8149-abb009967c25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:53.921105 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efe23782-c502-4028-8149-abb009967c25 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:53.921382 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efe23782-c502-4028-8149-abb009967c25 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:53.921662 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efe23782-c502-4028-8149-abb009967c25 None admin] PATH_INFO: `/v3/projects/a82de84c774f4bfb8eda1692f1d46a24` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:53.922765 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efe23782-c502-4028-8149-abb009967c25 None admin] RBAC: Authorizing `identity:get_project(project_id=a82de84c774f4bfb8eda1692f1d46a24)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:53.924264 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efe23782-c502-4028-8149-abb009967c25 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:53.930772 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 99/205] 10.222.0.77 () {56 vars in 1186 bytes} [Tue Aug 30 04:58:53 2022] GET /identity/v3/projects/a82de84c774f4bfb8eda1692f1d46a24 => generated 302 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:58:53.932009 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efe23782-c502-4028-8149-abb009967c25 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:53.937161 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8f3e1b2-8ea8-475c-aa26-9f6f9015ad1b None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:53.976305 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0acf5a12-3e9f-4df2-9f25-65e6b9fce353 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:53.977309 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0acf5a12-3e9f-4df2-9f25-65e6b9fce353 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:53.977863 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0acf5a12-3e9f-4df2-9f25-65e6b9fce353 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:53.978345 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0acf5a12-3e9f-4df2-9f25-65e6b9fce353 None admin] PATH_INFO: `/v3/projects/a82de84c774f4bfb8eda1692f1d46a24` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:53.980103 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0acf5a12-3e9f-4df2-9f25-65e6b9fce353 None admin] RBAC: Authorizing `identity:get_project(project_id=a82de84c774f4bfb8eda1692f1d46a24)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:53.981788 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0acf5a12-3e9f-4df2-9f25-65e6b9fce353 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:53.984173 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 107/206] 10.222.0.77 () {56 vars in 1186 bytes} [Tue Aug 30 04:58:53 2022] GET /identity/v3/projects/a82de84c774f4bfb8eda1692f1d46a24 => generated 302 bytes in 316 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:58:53.991901 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0acf5a12-3e9f-4df2-9f25-65e6b9fce353 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:54.035260 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efe23782-c502-4028-8149-abb009967c25 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:54.069389 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5930aa36-863f-4c77-a94b-8b87909b795c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:54.069992 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5930aa36-863f-4c77-a94b-8b87909b795c None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:54.070270 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5930aa36-863f-4c77-a94b-8b87909b795c None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:54.070550 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5930aa36-863f-4c77-a94b-8b87909b795c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:54.071903 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5930aa36-863f-4c77-a94b-8b87909b795c None admin] RBAC: Adding query filter params (role.id=3b9d5509a2f7478c9b4989d921daef2a, scope.project.id=a82de84c774f4bfb8eda1692f1d46a24, user.id=212278f6393840e1848128c611251d64) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:58:54.072207 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5930aa36-863f-4c77-a94b-8b87909b795c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:54.073643 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5930aa36-863f-4c77-a94b-8b87909b795c None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:54.079253 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 100/207] 10.222.0.77 () {56 vars in 1374 bytes} [Tue Aug 30 04:58:53 2022] GET /identity/v3/role_assignments?user.id=212278f6393840e1848128c611251d64&scope.project.id=a82de84c774f4bfb8eda1692f1d46a24&role.id=3b9d5509a2f7478c9b4989d921daef2a => generated 259 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:58:54.112956 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0acf5a12-3e9f-4df2-9f25-65e6b9fce353 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:54.154107 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc3de378-5204-427d-8c6b-2f61a2cac783 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:54.155133 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc3de378-5204-427d-8c6b-2f61a2cac783 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:54.155611 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc3de378-5204-427d-8c6b-2f61a2cac783 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:54.156107 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc3de378-5204-427d-8c6b-2f61a2cac783 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:54.157056 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc3de378-5204-427d-8c6b-2f61a2cac783 None admin] RBAC: Adding query filter params (role.id=9d9d5d2c270146488249b7ed943280db, scope.project.id=a82de84c774f4bfb8eda1692f1d46a24, user.id=212278f6393840e1848128c611251d64) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:58:54.157584 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc3de378-5204-427d-8c6b-2f61a2cac783 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:54.159273 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc3de378-5204-427d-8c6b-2f61a2cac783 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:54.166332 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 108/208] 10.222.0.77 () {56 vars in 1374 bytes} [Tue Aug 30 04:58:53 2022] GET /identity/v3/role_assignments?user.id=212278f6393840e1848128c611251d64&scope.project.id=a82de84c774f4bfb8eda1692f1d46a24&role.id=9d9d5d2c270146488249b7ed943280db => generated 259 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:58:55.112149 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b7c5cc1-466c-4273-8798-f57a366b71b6 None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:55.112654 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b7c5cc1-466c-4273-8798-f57a366b71b6 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:55.112728 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b7c5cc1-466c-4273-8798-f57a366b71b6 None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:55.114314 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 101/209] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:58:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:58:55.119248 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66188efe-3888-4cea-8fc6-8f6d34b56f73 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:55.119678 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66188efe-3888-4cea-8fc6-8f6d34b56f73 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:55.119863 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66188efe-3888-4cea-8fc6-8f6d34b56f73 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:55.126335 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:55.139993 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-66188efe-3888-4cea-8fc6-8f6d34b56f73 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:58:55.141198 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-66188efe-3888-4cea-8fc6-8f6d34b56f73 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:58:55.213616 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 109/210] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:58:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:58:55.256131 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d3c3c15-d801-4290-898f-b7e1cdf30612 None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:55.256738 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d3c3c15-d801-4290-898f-b7e1cdf30612 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:55.257203 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d3c3c15-d801-4290-898f-b7e1cdf30612 None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:55.258800 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 102/211] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:58:55 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:58:55.264088 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4755b9c8-7896-4888-8d24-33f4deb72bf6 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:55.264599 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4755b9c8-7896-4888-8d24-33f4deb72bf6 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:55.265068 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4755b9c8-7896-4888-8d24-33f4deb72bf6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:55.284693 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-4755b9c8-7896-4888-8d24-33f4deb72bf6 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:58:55.286105 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-4755b9c8-7896-4888-8d24-33f4deb72bf6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:58:55.355121 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 110/212] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:58:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:58:55.487693 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d3c3c15-d801-4290-898f-b7e1cdf30612 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:55.512585 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:55.595564 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d3c3c15-d801-4290-898f-b7e1cdf30612 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:55.630549 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46728b8b-55e0-4f0d-8364-8ff2476d77b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:55.631546 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46728b8b-55e0-4f0d-8364-8ff2476d77b8 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:55.632039 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46728b8b-55e0-4f0d-8364-8ff2476d77b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:55.632510 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46728b8b-55e0-4f0d-8364-8ff2476d77b8 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:55.637613 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4755b9c8-7896-4888-8d24-33f4deb72bf6 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:55.646797 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46728b8b-55e0-4f0d-8364-8ff2476d77b8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:58:55.652819 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-46728b8b-55e0-4f0d-8364-8ff2476d77b8 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 30 04:58:55.652819 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:58:55.652819 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:58:55.652819 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:58:55.652819 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:58:55.652819 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:58:55.652819 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:58:55.652819 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:58:55.652819 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:58:55.652819 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:58:55.652819 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:58:55.652819 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:58:55.652819 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:58:55.652819 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:58:55.652819 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:58:55.652819 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:58:55.653687 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:58:55.653687 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:58:55.653687 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:58:55.653687 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:58:55.653687 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:58:55.653687 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:58:55.653687 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:58:55.653687 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:58:55.653687 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:58:55.653687 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:58:55.653687 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:58:55.653687 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 04:58:55.653687 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:58:55.653687 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:58:55.653687 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:58:55.653687 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:58:55.654470 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:58:55.654470 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:58:55.654470 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:58:55.654470 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:58:55.654470 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:58:55.654470 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:58:55.654470 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:58:55.654470 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:58:55.654470 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:58:55.654470 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:58:55.654470 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 30 04:58:55.654470 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 04:58:55.658955 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 103/213] 10.222.0.77 () {56 vars in 1099 bytes} [Tue Aug 30 04:58:55 2022] GET /identity/v3/roles/member => generated 84 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:58:55.659807 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46728b8b-55e0-4f0d-8364-8ff2476d77b8 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:55.708277 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:55.750421 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:55.753790 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4755b9c8-7896-4888-8d24-33f4deb72bf6 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:55.770589 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46728b8b-55e0-4f0d-8364-8ff2476d77b8 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:55.792292 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f76d5c1-001d-4af8-9627-7a8bf2291d71 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:55.793185 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f76d5c1-001d-4af8-9627-7a8bf2291d71 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:55.793655 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f76d5c1-001d-4af8-9627-7a8bf2291d71 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:55.794116 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f76d5c1-001d-4af8-9627-7a8bf2291d71 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:55.795547 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f76d5c1-001d-4af8-9627-7a8bf2291d71 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:55.797265 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f76d5c1-001d-4af8-9627-7a8bf2291d71 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:55.799304 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 111/214] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 04:58:55 2022] GET /identity/v3/domains/default => generated 199 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:58:55.804649 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f76d5c1-001d-4af8-9627-7a8bf2291d71 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:55.808742 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-009cf8ff-e8a4-43be-b08e-33d8aa81d452 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:55.809353 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-009cf8ff-e8a4-43be-b08e-33d8aa81d452 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:55.809630 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-009cf8ff-e8a4-43be-b08e-33d8aa81d452 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:55.809897 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-009cf8ff-e8a4-43be-b08e-33d8aa81d452 None admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:55.811169 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-009cf8ff-e8a4-43be-b08e-33d8aa81d452 None admin] RBAC: Adding query filter params (name=member) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:58:55.811169 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-009cf8ff-e8a4-43be-b08e-33d8aa81d452 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:55.812423 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-009cf8ff-e8a4-43be-b08e-33d8aa81d452 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:55.828197 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 104/215] 10.222.0.77 () {56 vars in 1101 bytes} [Tue Aug 30 04:58:55 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:58:55.835287 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-009cf8ff-e8a4-43be-b08e-33d8aa81d452 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:55.917156 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f76d5c1-001d-4af8-9627-7a8bf2291d71 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:55.948960 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:55.955693 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8452bef-302d-45ad-a497-a6e3df535c94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:55.956887 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8452bef-302d-45ad-a497-a6e3df535c94 None admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:55.957634 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8452bef-302d-45ad-a497-a6e3df535c94 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:55.958238 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8452bef-302d-45ad-a497-a6e3df535c94 None admin] PATH_INFO: `/v3/users` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:55.959078 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8452bef-302d-45ad-a497-a6e3df535c94 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:55.960667 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8452bef-302d-45ad-a497-a6e3df535c94 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:55.973945 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-009cf8ff-e8a4-43be-b08e-33d8aa81d452 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:55.985395 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 112/216] 10.222.0.77 () {60 vars in 1132 bytes} [Tue Aug 30 04:58:55 2022] POST /identity/v3/users => generated 297 bytes in 181 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 04:58:56.013460 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d309576-28c7-4919-83f6-fd9f2011a988 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:56.014018 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d309576-28c7-4919-83f6-fd9f2011a988 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:56.014339 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d309576-28c7-4919-83f6-fd9f2011a988 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:56.014635 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d309576-28c7-4919-83f6-fd9f2011a988 None admin] PATH_INFO: `/v3/users/f8503162a9e14946a7a7272c543757c9` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:56.030632 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d309576-28c7-4919-83f6-fd9f2011a988 None admin] RBAC: Authorizing `identity:get_user(user_id=f8503162a9e14946a7a7272c543757c9)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:56.032209 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d309576-28c7-4919-83f6-fd9f2011a988 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:56.034139 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 105/217] 10.222.0.77 () {56 vars in 1177 bytes} [Tue Aug 30 04:58:55 2022] GET /identity/v3/users/f8503162a9e14946a7a7272c543757c9 => generated 297 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:58:56.039950 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8452bef-302d-45ad-a497-a6e3df535c94 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:56.063990 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c0f124-8fc6-4374-91de-ddc3b7fdd379 None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:56.064295 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c0f124-8fc6-4374-91de-ddc3b7fdd379 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:56.064538 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c0f124-8fc6-4374-91de-ddc3b7fdd379 None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:56.065913 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 106/218] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:58:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 04:58:56.071990 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b99281e3-813f-4f97-97a7-3eee449e9b6a None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:56.072289 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b99281e3-813f-4f97-97a7-3eee449e9b6a None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:56.072559 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b99281e3-813f-4f97-97a7-3eee449e9b6a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:56.080708 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:56.095399 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-b99281e3-813f-4f97-97a7-3eee449e9b6a None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:58:56.096643 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-b99281e3-813f-4f97-97a7-3eee449e9b6a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:58:56.163561 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8452bef-302d-45ad-a497-a6e3df535c94 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:56.179215 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 107/219] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:58:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 30 04:58:56.181628 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6df2bca0-7590-4fea-bd6d-e1a54ec45b7d None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:56.181912 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6df2bca0-7590-4fea-bd6d-e1a54ec45b7d None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:56.182243 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6df2bca0-7590-4fea-bd6d-e1a54ec45b7d None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:56.183670 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 108/220] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:58:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:58:56.189569 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f05016-3512-4852-88b7-217d6746561b None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:56.189855 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f05016-3512-4852-88b7-217d6746561b None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:56.190127 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f05016-3512-4852-88b7-217d6746561b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:56.202583 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c17a564-cc40-44ca-b711-f5f4fd6a6c18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:56.203285 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c17a564-cc40-44ca-b711-f5f4fd6a6c18 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:56.204008 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c17a564-cc40-44ca-b711-f5f4fd6a6c18 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:56.204290 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c17a564-cc40-44ca-b711-f5f4fd6a6c18 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:56.209415 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-83f05016-3512-4852-88b7-217d6746561b None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:58:56.210601 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-83f05016-3512-4852-88b7-217d6746561b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:58:56.269425 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:56.294544 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c17a564-cc40-44ca-b711-f5f4fd6a6c18 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:56.296424 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c17a564-cc40-44ca-b711-f5f4fd6a6c18 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:56.299074 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 109/221] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:58:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:58:56.404338 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 113/222] 10.222.0.77 () {58 vars in 1282 bytes} [Tue Aug 30 04:58:56 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 365 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 04:58:56.410352 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83f05016-3512-4852-88b7-217d6746561b None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:56.469863 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c17a564-cc40-44ca-b711-f5f4fd6a6c18 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:56.478489 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:56.527679 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83f05016-3512-4852-88b7-217d6746561b None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:56.584535 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d2841d2-f609-42a3-8646-1e1b61f005a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:56.586301 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d2841d2-f609-42a3-8646-1e1b61f005a6 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:56.587492 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d2841d2-f609-42a3-8646-1e1b61f005a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:56.589163 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d2841d2-f609-42a3-8646-1e1b61f005a6 None admin] PATH_INFO: `/v3/projects/a82de84c774f4bfb8eda1692f1d46a24` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:56.590848 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:56.591994 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d2841d2-f609-42a3-8646-1e1b61f005a6 None admin] RBAC: Authorizing `identity:get_project(project_id=a82de84c774f4bfb8eda1692f1d46a24)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:56.596342 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d2841d2-f609-42a3-8646-1e1b61f005a6 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:56.599258 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 110/223] 10.222.0.77 () {56 vars in 1186 bytes} [Tue Aug 30 04:58:56 2022] GET /identity/v3/projects/a82de84c774f4bfb8eda1692f1d46a24 => generated 302 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:58:56.600467 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d2841d2-f609-42a3-8646-1e1b61f005a6 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:56.616522 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c17a564-cc40-44ca-b711-f5f4fd6a6c18 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:56.663345 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b537a39-9335-4271-8abb-6487dcd27aac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:56.664311 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b537a39-9335-4271-8abb-6487dcd27aac None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:56.664688 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b537a39-9335-4271-8abb-6487dcd27aac None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:56.665819 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b537a39-9335-4271-8abb-6487dcd27aac None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:56.679311 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b537a39-9335-4271-8abb-6487dcd27aac None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:58:56.685805 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-8b537a39-9335-4271-8abb-6487dcd27aac None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 04:58:56.685805 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:58:56.685805 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:58:56.685805 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:58:56.685805 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:58:56.685805 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:58:56.685805 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:58:56.685805 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:58:56.685805 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:58:56.685805 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:58:56.685805 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:58:56.685805 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:58:56.685805 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:58:56.685805 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:58:56.685805 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:58:56.685805 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:58:56.686731 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:58:56.686731 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:58:56.686731 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:58:56.686731 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:58:56.686731 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:58:56.686731 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:58:56.686731 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:58:56.686731 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:58:56.686731 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:58:56.686731 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:58:56.686731 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:58:56.686731 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 04:58:56.686731 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:58:56.686731 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:58:56.686731 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:58:56.686731 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:58:56.687552 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:58:56.687552 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:58:56.687552 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:58:56.687552 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:58:56.687552 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:58:56.687552 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:58:56.687552 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:58:56.687552 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:58:56.687552 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:58:56.687552 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:58:56.687552 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 04:58:56.687552 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 04:58:56.687552 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 114/224] 10.222.0.77 () {56 vars in 1096 bytes} [Tue Aug 30 04:58:56 2022] GET /identity/v3/roles/admin => generated 83 bytes in 218 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 04:58:56.688269 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b537a39-9335-4271-8abb-6487dcd27aac None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:56.726296 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d2841d2-f609-42a3-8646-1e1b61f005a6 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:56.769837 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:56.780995 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eef0aee4-d519-43c2-bafd-d62df378925b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:56.781560 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef0aee4-d519-43c2-bafd-d62df378925b None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:56.781835 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef0aee4-d519-43c2-bafd-d62df378925b None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:56.782102 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef0aee4-d519-43c2-bafd-d62df378925b None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:56.802119 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eef0aee4-d519-43c2-bafd-d62df378925b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:58:56.808456 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b537a39-9335-4271-8abb-6487dcd27aac None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:56.809430 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-eef0aee4-d519-43c2-bafd-d62df378925b None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 30 04:58:56.809430 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:58:56.809430 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:58:56.809430 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:58:56.809430 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:58:56.809430 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:58:56.809430 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:58:56.809430 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:58:56.809430 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:58:56.809430 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:58:56.809430 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:58:56.809430 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:58:56.809430 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:58:56.809430 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:58:56.809430 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:58:56.809430 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:58:56.810262 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:58:56.810262 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:58:56.810262 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:58:56.810262 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:58:56.810262 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:58:56.810262 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:58:56.810262 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:58:56.810262 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:58:56.810262 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:58:56.810262 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:58:56.810262 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:58:56.810262 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 04:58:56.810262 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:58:56.810262 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:58:56.810262 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:58:56.810262 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:58:56.811091 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:58:56.811091 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:58:56.811091 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:58:56.811091 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:58:56.811091 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:58:56.811091 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:58:56.811091 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:58:56.811091 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:58:56.811091 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:58:56.811091 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:58:56.811091 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 30 04:58:56.811091 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 04:58:56.812288 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 111/225] 10.222.0.77 () {56 vars in 1114 bytes} [Tue Aug 30 04:58:56 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:58:56.813070 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eef0aee4-d519-43c2-bafd-d62df378925b None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:56.850419 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f689670-ddca-435c-908e-ea33248891b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:56.851496 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f689670-ddca-435c-908e-ea33248891b8 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:56.851983 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f689670-ddca-435c-908e-ea33248891b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:56.853038 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f689670-ddca-435c-908e-ea33248891b8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:56.854316 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f689670-ddca-435c-908e-ea33248891b8 None admin] RBAC: Adding query filter params (role.id=6f03d86980c34c36b44a8d6764ff4fda, scope.project.id=a82de84c774f4bfb8eda1692f1d46a24, user.id=f8503162a9e14946a7a7272c543757c9) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:58:56.854834 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f689670-ddca-435c-908e-ea33248891b8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:56.856559 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f689670-ddca-435c-908e-ea33248891b8 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:56.862710 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 115/226] 10.222.0.77 () {56 vars in 1374 bytes} [Tue Aug 30 04:58:56 2022] GET /identity/v3/role_assignments?user.id=f8503162a9e14946a7a7272c543757c9&scope.project.id=a82de84c774f4bfb8eda1692f1d46a24&role.id=6f03d86980c34c36b44a8d6764ff4fda => generated 259 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:58:56.872342 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f689670-ddca-435c-908e-ea33248891b8 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:56.885777 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:56.928530 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eef0aee4-d519-43c2-bafd-d62df378925b None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:56.975518 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a147582-dc97-4b71-81bb-a40c5dba22c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:56.976142 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a147582-dc97-4b71-81bb-a40c5dba22c9 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:56.976421 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a147582-dc97-4b71-81bb-a40c5dba22c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:56.976690 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a147582-dc97-4b71-81bb-a40c5dba22c9 None admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:56.977420 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a147582-dc97-4b71-81bb-a40c5dba22c9 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:58:56.977724 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a147582-dc97-4b71-81bb-a40c5dba22c9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:56.979454 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a147582-dc97-4b71-81bb-a40c5dba22c9 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:56.996070 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 112/227] 10.222.0.77 () {56 vars in 1099 bytes} [Tue Aug 30 04:58:56 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:58:57.003274 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a147582-dc97-4b71-81bb-a40c5dba22c9 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:57.082700 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f689670-ddca-435c-908e-ea33248891b8 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:57.094916 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:57.121295 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c086ff63-c4be-4983-b6ea-1af2a13fab07 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:57.122062 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c086ff63-c4be-4983-b6ea-1af2a13fab07 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:57.122628 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c086ff63-c4be-4983-b6ea-1af2a13fab07 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:57.122746 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c086ff63-c4be-4983-b6ea-1af2a13fab07 None admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:57.123546 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c086ff63-c4be-4983-b6ea-1af2a13fab07 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:58:57.123937 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c086ff63-c4be-4983-b6ea-1af2a13fab07 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:57.125413 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c086ff63-c4be-4983-b6ea-1af2a13fab07 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:57.134633 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 116/228] 10.222.0.77 () {56 vars in 1111 bytes} [Tue Aug 30 04:58:56 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 263 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:58:57.143989 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c086ff63-c4be-4983-b6ea-1af2a13fab07 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:57.149220 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a147582-dc97-4b71-81bb-a40c5dba22c9 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:57.224393 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:57.245695 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b678aae1-f13e-48ae-b3ab-4149c5184e82 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:57.247013 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b678aae1-f13e-48ae-b3ab-4149c5184e82 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:57.250175 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b678aae1-f13e-48ae-b3ab-4149c5184e82 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:57.250654 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b678aae1-f13e-48ae-b3ab-4149c5184e82 None admin] PATH_INFO: `/v3/users/212278f6393840e1848128c611251d64` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:57.252972 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b678aae1-f13e-48ae-b3ab-4149c5184e82 None admin] RBAC: Authorizing `identity:get_user(user_id=212278f6393840e1848128c611251d64)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:57.254725 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b678aae1-f13e-48ae-b3ab-4149c5184e82 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:57.256705 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 113/229] 10.222.0.77 () {56 vars in 1177 bytes} [Tue Aug 30 04:58:57 2022] GET /identity/v3/users/212278f6393840e1848128c611251d64 => generated 283 bytes in 254 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:58:57.263753 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b678aae1-f13e-48ae-b3ab-4149c5184e82 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:57.279597 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c086ff63-c4be-4983-b6ea-1af2a13fab07 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:57.347999 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4e27c7a-4326-4de7-ad10-fcf69946cc38 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:57.348967 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4e27c7a-4326-4de7-ad10-fcf69946cc38 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:57.350535 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4e27c7a-4326-4de7-ad10-fcf69946cc38 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:57.351038 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4e27c7a-4326-4de7-ad10-fcf69946cc38 None admin] PATH_INFO: `/v3/users/212278f6393840e1848128c611251d64` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:57.352475 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4e27c7a-4326-4de7-ad10-fcf69946cc38 None admin] RBAC: Authorizing `identity:get_user(user_id=212278f6393840e1848128c611251d64)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:57.354515 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4e27c7a-4326-4de7-ad10-fcf69946cc38 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:57.356995 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 117/230] 10.222.0.77 () {56 vars in 1177 bytes} [Tue Aug 30 04:58:57 2022] GET /identity/v3/users/212278f6393840e1848128c611251d64 => generated 283 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:58:57.363160 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4e27c7a-4326-4de7-ad10-fcf69946cc38 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:57.489403 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b678aae1-f13e-48ae-b3ab-4149c5184e82 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:57.518631 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4e27c7a-4326-4de7-ad10-fcf69946cc38 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:57.525086 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:57.555919 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8db879ea-fc6b-4544-b905-67c2bc7ff817 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:57.556583 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8db879ea-fc6b-4544-b905-67c2bc7ff817 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:57.556862 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8db879ea-fc6b-4544-b905-67c2bc7ff817 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:57.557130 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8db879ea-fc6b-4544-b905-67c2bc7ff817 None admin] PATH_INFO: `/v3/projects/a82de84c774f4bfb8eda1692f1d46a24` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:57.559809 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8db879ea-fc6b-4544-b905-67c2bc7ff817 None admin] RBAC: Authorizing `identity:get_project(project_id=a82de84c774f4bfb8eda1692f1d46a24)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:57.561322 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8db879ea-fc6b-4544-b905-67c2bc7ff817 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:57.563797 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 114/231] 10.222.0.77 () {56 vars in 1186 bytes} [Tue Aug 30 04:58:57 2022] GET /identity/v3/projects/a82de84c774f4bfb8eda1692f1d46a24 => generated 302 bytes in 300 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:58:57.571896 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8db879ea-fc6b-4544-b905-67c2bc7ff817 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:57.583779 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7d9a372-29a4-43ed-82c9-8031269d9039 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:57.584414 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7d9a372-29a4-43ed-82c9-8031269d9039 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:57.584690 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7d9a372-29a4-43ed-82c9-8031269d9039 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:57.584958 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7d9a372-29a4-43ed-82c9-8031269d9039 None admin] PATH_INFO: `/v3/projects/a82de84c774f4bfb8eda1692f1d46a24` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:57.586290 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7d9a372-29a4-43ed-82c9-8031269d9039 None admin] RBAC: Authorizing `identity:get_project(project_id=a82de84c774f4bfb8eda1692f1d46a24)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:57.587817 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7d9a372-29a4-43ed-82c9-8031269d9039 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:57.591684 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 118/232] 10.222.0.77 () {56 vars in 1186 bytes} [Tue Aug 30 04:58:57 2022] GET /identity/v3/projects/a82de84c774f4bfb8eda1692f1d46a24 => generated 302 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:58:57.598254 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7d9a372-29a4-43ed-82c9-8031269d9039 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:57.621092 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:57.758318 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8db879ea-fc6b-4544-b905-67c2bc7ff817 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:57.779860 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7d9a372-29a4-43ed-82c9-8031269d9039 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:57.821179 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b5134a5-4f78-453c-9818-d2908b9b7c12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:57.821964 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b5134a5-4f78-453c-9818-d2908b9b7c12 None admin] REQUEST_METHOD: `PUT` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:57.822289 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b5134a5-4f78-453c-9818-d2908b9b7c12 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:57.822564 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b5134a5-4f78-453c-9818-d2908b9b7c12 None admin] PATH_INFO: `/v3/projects/a82de84c774f4bfb8eda1692f1d46a24/users/212278f6393840e1848128c611251d64/roles/9d9d5d2c270146488249b7ed943280db` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:57.826210 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b5134a5-4f78-453c-9818-d2908b9b7c12 None admin] RBAC: Authorizing `identity:create_grant(project_id=a82de84c774f4bfb8eda1692f1d46a24, user_id=212278f6393840e1848128c611251d64, role_id=9d9d5d2c270146488249b7ed943280db)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:57.827677 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b5134a5-4f78-453c-9818-d2908b9b7c12 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:57.842928 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 115/233] 10.222.0.77 () {58 vars in 1439 bytes} [Tue Aug 30 04:58:57 2022] PUT /identity/v3/projects/a82de84c774f4bfb8eda1692f1d46a24/users/212278f6393840e1848128c611251d64/roles/9d9d5d2c270146488249b7ed943280db => generated 0 bytes in 272 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 04:58:57.847506 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc15c7c9-dbc3-495f-be53-afbe72ade2e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:57.848233 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc15c7c9-dbc3-495f-be53-afbe72ade2e2 None admin] REQUEST_METHOD: `PUT` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:57.848615 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc15c7c9-dbc3-495f-be53-afbe72ade2e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:57.848886 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc15c7c9-dbc3-495f-be53-afbe72ade2e2 None admin] PATH_INFO: `/v3/projects/a82de84c774f4bfb8eda1692f1d46a24/users/212278f6393840e1848128c611251d64/roles/3b9d5509a2f7478c9b4989d921daef2a` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:57.851255 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc15c7c9-dbc3-495f-be53-afbe72ade2e2 None admin] RBAC: Authorizing `identity:create_grant(project_id=a82de84c774f4bfb8eda1692f1d46a24, user_id=212278f6393840e1848128c611251d64, role_id=3b9d5509a2f7478c9b4989d921daef2a)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:57.853554 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc15c7c9-dbc3-495f-be53-afbe72ade2e2 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:57.885120 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 119/234] 10.222.0.77 () {58 vars in 1439 bytes} [Tue Aug 30 04:58:57 2022] PUT /identity/v3/projects/a82de84c774f4bfb8eda1692f1d46a24/users/212278f6393840e1848128c611251d64/roles/3b9d5509a2f7478c9b4989d921daef2a => generated 0 bytes in 287 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:58:58.336673 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f6669f9-a217-42c5-b53b-ea4343eabb6e None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:58.337697 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f6669f9-a217-42c5-b53b-ea4343eabb6e None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:58.337772 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f6669f9-a217-42c5-b53b-ea4343eabb6e None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:58.339265 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 116/235] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:58:58 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:58:58.348693 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b914fdf-42c7-46e1-9bbd-285e7cfff29b None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:58.349154 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b914fdf-42c7-46e1-9bbd-285e7cfff29b None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:58.349231 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b914fdf-42c7-46e1-9bbd-285e7cfff29b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:58.377778 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-5b914fdf-42c7-46e1-9bbd-285e7cfff29b None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:58:58.378992 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-5b914fdf-42c7-46e1-9bbd-285e7cfff29b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:58:58.442165 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f82f56d4-675b-4fb9-9b0b-500d76f53d60 None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:58.442519 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f82f56d4-675b-4fb9-9b0b-500d76f53d60 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:58.442717 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f82f56d4-675b-4fb9-9b0b-500d76f53d60 None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:58.444242 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 117/236] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:58:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:58:58.459144 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac1658f-1ce5-406a-9330-f6de74a285b0 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:58.459419 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac1658f-1ce5-406a-9330-f6de74a285b0 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:58.459684 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac1658f-1ce5-406a-9330-f6de74a285b0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:58.483762 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:58.487120 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 120/237] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:58:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:58:58.499842 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-5ac1658f-1ce5-406a-9330-f6de74a285b0 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:58:58.501073 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-5ac1658f-1ce5-406a-9330-f6de74a285b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:58:58.553673 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:58.584692 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 118/238] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:58:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 30 04:58:58.823781 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b914fdf-42c7-46e1-9bbd-285e7cfff29b None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:58.902150 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ac1658f-1ce5-406a-9330-f6de74a285b0 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:58.932132 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b914fdf-42c7-46e1-9bbd-285e7cfff29b None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:58.966628 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3deb0dc8-fe7d-4bc6-a226-408e704ce949 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:58.967281 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3deb0dc8-fe7d-4bc6-a226-408e704ce949 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:58.967560 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3deb0dc8-fe7d-4bc6-a226-408e704ce949 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:58.967827 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3deb0dc8-fe7d-4bc6-a226-408e704ce949 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:58.978061 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3deb0dc8-fe7d-4bc6-a226-408e704ce949 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:58:58.983216 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-3deb0dc8-fe7d-4bc6-a226-408e704ce949 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 30 04:58:58.983216 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:58:58.983216 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:58:58.983216 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:58:58.983216 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:58:58.983216 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:58:58.983216 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:58:58.983216 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:58:58.983216 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:58:58.983216 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:58:58.983216 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:58:58.983216 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:58:58.983216 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:58:58.983216 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:58:58.983216 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:58:58.983216 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:58:58.985242 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:58:58.985242 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:58:58.985242 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:58:58.985242 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:58:58.985242 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:58:58.985242 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:58:58.985242 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:58:58.985242 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:58:58.985242 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:58:58.985242 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:58:58.985242 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:58:58.985242 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 04:58:58.985242 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:58:58.985242 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:58:58.985242 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:58:58.985242 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:58:58.987254 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:58:58.987254 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:58:58.987254 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:58:58.987254 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:58:58.987254 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:58:58.987254 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:58:58.987254 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:58:58.987254 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:58:58.987254 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:58:58.987254 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:58:58.987254 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 30 04:58:58.987254 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 04:58:58.995098 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 121/239] 10.222.0.77 () {56 vars in 1099 bytes} [Tue Aug 30 04:58:58 2022] GET /identity/v3/roles/reader => generated 84 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:58:58.999697 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3deb0dc8-fe7d-4bc6-a226-408e704ce949 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:59.008613 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ac1658f-1ce5-406a-9330-f6de74a285b0 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:59.046114 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40cd7589-c044-48af-80f2-5841d85e82f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:59.046871 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40cd7589-c044-48af-80f2-5841d85e82f1 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:59.047158 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40cd7589-c044-48af-80f2-5841d85e82f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:59.047585 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40cd7589-c044-48af-80f2-5841d85e82f1 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:59.049458 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40cd7589-c044-48af-80f2-5841d85e82f1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:59.050950 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40cd7589-c044-48af-80f2-5841d85e82f1 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:59.052752 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 119/240] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 04:58:58 2022] GET /identity/v3/domains/default => generated 199 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:58:59.058964 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40cd7589-c044-48af-80f2-5841d85e82f1 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:59.111048 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:59.147177 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:59.173006 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40cd7589-c044-48af-80f2-5841d85e82f1 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:59.213839 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e748e2b-aa52-4471-8d58-80bf7e53eb47 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:59.214512 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e748e2b-aa52-4471-8d58-80bf7e53eb47 None admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:59.214795 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e748e2b-aa52-4471-8d58-80bf7e53eb47 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:59.215085 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e748e2b-aa52-4471-8d58-80bf7e53eb47 None admin] PATH_INFO: `/v3/users` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:59.215737 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e748e2b-aa52-4471-8d58-80bf7e53eb47 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:59.217173 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e748e2b-aa52-4471-8d58-80bf7e53eb47 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:59.241681 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 120/241] 10.222.0.77 () {60 vars in 1132 bytes} [Tue Aug 30 04:58:59 2022] POST /identity/v3/users => generated 293 bytes in 183 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 04:58:59.271503 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3deb0dc8-fe7d-4bc6-a226-408e704ce949 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:59.314876 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0008378c-5d55-4ddf-a2d2-ba1d431d04a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:59.315507 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0008378c-5d55-4ddf-a2d2-ba1d431d04a4 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:59.315784 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0008378c-5d55-4ddf-a2d2-ba1d431d04a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:59.316053 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0008378c-5d55-4ddf-a2d2-ba1d431d04a4 None admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:59.316731 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0008378c-5d55-4ddf-a2d2-ba1d431d04a4 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:58:59.317030 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0008378c-5d55-4ddf-a2d2-ba1d431d04a4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:59.318430 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0008378c-5d55-4ddf-a2d2-ba1d431d04a4 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:59.333522 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 122/242] 10.222.0.77 () {56 vars in 1101 bytes} [Tue Aug 30 04:58:58 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 334 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:58:59.339309 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e748e2b-aa52-4471-8d58-80bf7e53eb47 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:59.417249 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66775698-c75f-4e03-b939-db099f6e742e None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:59.417754 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66775698-c75f-4e03-b939-db099f6e742e None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:59.417754 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66775698-c75f-4e03-b939-db099f6e742e None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:59.419386 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 123/243] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:58:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 04:58:59.434547 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79623092-3606-4a2b-b63d-5c958bce1f25 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:59.434937 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79623092-3606-4a2b-b63d-5c958bce1f25 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:59.435131 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79623092-3606-4a2b-b63d-5c958bce1f25 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:59.473964 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-79623092-3606-4a2b-b63d-5c958bce1f25 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:58:59.476181 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-79623092-3606-4a2b-b63d-5c958bce1f25 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:58:59.494743 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e748e2b-aa52-4471-8d58-80bf7e53eb47 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:59.496676 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:59.539275 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d127c802-3c2c-45e2-a205-07d4b45b6e5c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:59.541389 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d127c802-3c2c-45e2-a205-07d4b45b6e5c None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:59.541917 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d127c802-3c2c-45e2-a205-07d4b45b6e5c None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:59.542286 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d127c802-3c2c-45e2-a205-07d4b45b6e5c None admin] PATH_INFO: `/v3/users/9f1b91507f9243b5bcd11759ab3a4592` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:59.557756 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d127c802-3c2c-45e2-a205-07d4b45b6e5c None admin] RBAC: Authorizing `identity:get_user(user_id=9f1b91507f9243b5bcd11759ab3a4592)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:59.559708 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d127c802-3c2c-45e2-a205-07d4b45b6e5c None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:59.561353 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 121/244] 10.222.0.77 () {56 vars in 1177 bytes} [Tue Aug 30 04:58:59 2022] GET /identity/v3/users/9f1b91507f9243b5bcd11759ab3a4592 => generated 297 bytes in 223 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:58:59.567430 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d127c802-3c2c-45e2-a205-07d4b45b6e5c None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:59.576188 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 124/245] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:58:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 30 04:58:59.672156 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d127c802-3c2c-45e2-a205-07d4b45b6e5c None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:58:59.706671 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-742a9686-ab8a-4d5c-b8b1-0aa8a0c54ea0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:58:59.707318 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-742a9686-ab8a-4d5c-b8b1-0aa8a0c54ea0 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:58:59.707594 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-742a9686-ab8a-4d5c-b8b1-0aa8a0c54ea0 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:58:59.707862 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-742a9686-ab8a-4d5c-b8b1-0aa8a0c54ea0 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:58:59.738325 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:59.778894 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-742a9686-ab8a-4d5c-b8b1-0aa8a0c54ea0 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:58:59.780374 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-742a9686-ab8a-4d5c-b8b1-0aa8a0c54ea0 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:58:59.882632 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 122/246] 10.222.0.77 () {58 vars in 1282 bytes} [Tue Aug 30 04:58:59 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 316 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Aug 30 04:58:59.889681 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79623092-3606-4a2b-b63d-5c958bce1f25 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:59.910288 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-742a9686-ab8a-4d5c-b8b1-0aa8a0c54ea0 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:58:59.953980 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:58:59.976899 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:00.013646 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79623092-3606-4a2b-b63d-5c958bce1f25 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:00.022281 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-742a9686-ab8a-4d5c-b8b1-0aa8a0c54ea0 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:00.052192 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aabbb09-628f-404f-bd7d-4f4e9d8526a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:00.053272 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aabbb09-628f-404f-bd7d-4f4e9d8526a6 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:00.053836 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aabbb09-628f-404f-bd7d-4f4e9d8526a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:00.054318 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aabbb09-628f-404f-bd7d-4f4e9d8526a6 None admin] PATH_INFO: `/v3/projects/a82de84c774f4bfb8eda1692f1d46a24` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:00.055892 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aabbb09-628f-404f-bd7d-4f4e9d8526a6 None admin] RBAC: Authorizing `identity:get_project(project_id=a82de84c774f4bfb8eda1692f1d46a24)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:00.058053 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aabbb09-628f-404f-bd7d-4f4e9d8526a6 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:00.058530 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcd7fd92-e22b-4d87-a198-9f21ff7af41d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:00.059188 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd7fd92-e22b-4d87-a198-9f21ff7af41d None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:00.059464 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd7fd92-e22b-4d87-a198-9f21ff7af41d None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:00.059729 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd7fd92-e22b-4d87-a198-9f21ff7af41d None admin] PATH_INFO: `/v3/roles/member` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:00.061302 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 125/247] 10.222.0.77 () {56 vars in 1186 bytes} [Tue Aug 30 04:58:59 2022] GET /identity/v3/projects/a82de84c774f4bfb8eda1692f1d46a24 => generated 302 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:00.069574 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aabbb09-628f-404f-bd7d-4f4e9d8526a6 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:00.083846 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcd7fd92-e22b-4d87-a198-9f21ff7af41d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:59:00.091434 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-bcd7fd92-e22b-4d87-a198-9f21ff7af41d None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 30 04:59:00.091434 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:00.091434 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:00.091434 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:00.091434 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:00.091434 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:00.091434 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:00.091434 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:00.091434 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:00.091434 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:00.091434 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:00.091434 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:00.091434 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:59:00.091434 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:59:00.091434 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:59:00.091434 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:59:00.092315 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:59:00.092315 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:59:00.092315 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:59:00.092315 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:59:00.092315 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:00.092315 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:00.092315 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:00.092315 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:00.092315 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:00.092315 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:00.092315 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:00.092315 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:00.092315 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:00.092315 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:00.092315 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:00.092315 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:00.093091 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:00.093091 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:00.093091 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:00.093091 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:00.093091 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:59:00.093091 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:59:00.093091 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:59:00.093091 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:59:00.093091 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:59:00.093091 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:59:00.093091 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 30 04:59:00.093091 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 04:59:00.095538 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 123/248] 10.222.0.77 () {56 vars in 1099 bytes} [Tue Aug 30 04:58:59 2022] GET /identity/v3/roles/member => generated 84 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:00.101926 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcd7fd92-e22b-4d87-a198-9f21ff7af41d None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:00.220880 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aabbb09-628f-404f-bd7d-4f4e9d8526a6 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:00.246472 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:00.247925 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcd7fd92-e22b-4d87-a198-9f21ff7af41d None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:00.268227 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c72f45ba-35e5-4930-af8d-c9c338b00e5e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:00.268838 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c72f45ba-35e5-4930-af8d-c9c338b00e5e None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:00.269114 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c72f45ba-35e5-4930-af8d-c9c338b00e5e None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:00.269402 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c72f45ba-35e5-4930-af8d-c9c338b00e5e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:00.270312 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c72f45ba-35e5-4930-af8d-c9c338b00e5e None admin] RBAC: Adding query filter params (role.id=ddaf2963f700481eb93adea7a36f4e73, scope.project.id=a82de84c774f4bfb8eda1692f1d46a24, user.id=9f1b91507f9243b5bcd11759ab3a4592) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:00.270626 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c72f45ba-35e5-4930-af8d-c9c338b00e5e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:00.272812 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c72f45ba-35e5-4930-af8d-c9c338b00e5e None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:00.280591 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:00.281427 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 126/249] 10.222.0.77 () {56 vars in 1374 bytes} [Tue Aug 30 04:59:00 2022] GET /identity/v3/role_assignments?user.id=9f1b91507f9243b5bcd11759ab3a4592&scope.project.id=a82de84c774f4bfb8eda1692f1d46a24&role.id=ddaf2963f700481eb93adea7a36f4e73 => generated 259 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:00.283018 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1114b34b-3e7a-4639-8742-cfe5702725ae None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:00.283501 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1114b34b-3e7a-4639-8742-cfe5702725ae None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:00.287308 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1114b34b-3e7a-4639-8742-cfe5702725ae None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:00.290885 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 127/250] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:59:00 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:59:00.297762 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d30fc96f-7c8c-412e-a727-9a95d3651771 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:00.298374 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d30fc96f-7c8c-412e-a727-9a95d3651771 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:00.298913 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d30fc96f-7c8c-412e-a727-9a95d3651771 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:00.323615 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd7605fd-864c-4792-9847-74b64d653ab4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:00.324245 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd7605fd-864c-4792-9847-74b64d653ab4 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:00.324522 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd7605fd-864c-4792-9847-74b64d653ab4 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:00.324805 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd7605fd-864c-4792-9847-74b64d653ab4 None admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:00.325468 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd7605fd-864c-4792-9847-74b64d653ab4 None admin] RBAC: Adding query filter params (name=member) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:00.325767 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd7605fd-864c-4792-9847-74b64d653ab4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:00.327193 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd7605fd-864c-4792-9847-74b64d653ab4 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:00.331436 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-d30fc96f-7c8c-412e-a727-9a95d3651771 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:59:00.333464 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-d30fc96f-7c8c-412e-a727-9a95d3651771 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:59:00.336556 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 124/251] 10.222.0.77 () {56 vars in 1101 bytes} [Tue Aug 30 04:59:00 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 235 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:59:00.344907 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd7605fd-864c-4792-9847-74b64d653ab4 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:00.447881 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 128/252] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:59:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:59:00.468753 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ec3327-fec9-4c66-9e17-935f6ed6966a None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:00.469036 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ec3327-fec9-4c66-9e17-935f6ed6966a None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:00.469311 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ec3327-fec9-4c66-9e17-935f6ed6966a None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:00.470694 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 129/253] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:59:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 04:59:00.476329 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-604a151c-c825-48c7-895a-233b8bbbbccd None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:00.476608 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-604a151c-c825-48c7-895a-233b8bbbbccd None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:00.476791 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-604a151c-c825-48c7-895a-233b8bbbbccd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:00.505994 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-604a151c-c825-48c7-895a-233b8bbbbccd None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:59:00.507516 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-604a151c-c825-48c7-895a-233b8bbbbccd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:59:00.531656 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd7605fd-864c-4792-9847-74b64d653ab4 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:00.595014 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 130/254] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:59:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 30 04:59:00.600435 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac943c3f-ad20-4280-bc35-d54815a168a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:00.601067 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac943c3f-ad20-4280-bc35-d54815a168a3 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:00.601348 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac943c3f-ad20-4280-bc35-d54815a168a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:00.601616 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac943c3f-ad20-4280-bc35-d54815a168a3 None admin] PATH_INFO: `/v3/users/f8503162a9e14946a7a7272c543757c9` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:00.604808 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac943c3f-ad20-4280-bc35-d54815a168a3 None admin] RBAC: Authorizing `identity:get_user(user_id=f8503162a9e14946a7a7272c543757c9)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:00.606280 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac943c3f-ad20-4280-bc35-d54815a168a3 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:00.608313 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 125/255] 10.222.0.77 () {56 vars in 1177 bytes} [Tue Aug 30 04:59:00 2022] GET /identity/v3/users/f8503162a9e14946a7a7272c543757c9 => generated 297 bytes in 264 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:59:00.613645 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-604a151c-c825-48c7-895a-233b8bbbbccd None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:00.686421 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:00.759049 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-604a151c-c825-48c7-895a-233b8bbbbccd None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:00.801597 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bb4c004-7afe-4caf-a751-698869e323a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:00.802229 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb4c004-7afe-4caf-a751-698869e323a3 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:00.802507 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb4c004-7afe-4caf-a751-698869e323a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:00.802780 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb4c004-7afe-4caf-a751-698869e323a3 None admin] PATH_INFO: `/v3/projects/a82de84c774f4bfb8eda1692f1d46a24` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:00.803911 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bb4c004-7afe-4caf-a751-698869e323a3 None admin] RBAC: Authorizing `identity:get_project(project_id=a82de84c774f4bfb8eda1692f1d46a24)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:00.805910 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bb4c004-7afe-4caf-a751-698869e323a3 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:00.807733 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 131/256] 10.222.0.77 () {56 vars in 1186 bytes} [Tue Aug 30 04:59:00 2022] GET /identity/v3/projects/a82de84c774f4bfb8eda1692f1d46a24 => generated 302 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:00.813500 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac943c3f-ad20-4280-bc35-d54815a168a3 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:00.879877 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:00.935104 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac943c3f-ad20-4280-bc35-d54815a168a3 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:00.969301 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8248ccb-ddd0-4166-ba9c-716d700d1e43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:00.969958 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8248ccb-ddd0-4166-ba9c-716d700d1e43 None admin] REQUEST_METHOD: `PUT` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:00.970237 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8248ccb-ddd0-4166-ba9c-716d700d1e43 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:00.970509 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8248ccb-ddd0-4166-ba9c-716d700d1e43 None admin] PATH_INFO: `/v3/projects/a82de84c774f4bfb8eda1692f1d46a24/users/f8503162a9e14946a7a7272c543757c9/roles/6f03d86980c34c36b44a8d6764ff4fda` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:00.972506 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8248ccb-ddd0-4166-ba9c-716d700d1e43 None admin] RBAC: Authorizing `identity:create_grant(project_id=a82de84c774f4bfb8eda1692f1d46a24, user_id=f8503162a9e14946a7a7272c543757c9, role_id=6f03d86980c34c36b44a8d6764ff4fda)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:00.973964 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8248ccb-ddd0-4166-ba9c-716d700d1e43 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:00.984678 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bb4c004-7afe-4caf-a751-698869e323a3 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:00.993892 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 126/257] 10.222.0.77 () {58 vars in 1439 bytes} [Tue Aug 30 04:59:00 2022] PUT /identity/v3/projects/a82de84c774f4bfb8eda1692f1d46a24/users/f8503162a9e14946a7a7272c543757c9/roles/6f03d86980c34c36b44a8d6764ff4fda => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:59:01.073484 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8248ccb-ddd0-4166-ba9c-716d700d1e43 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:01.125292 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bb4c004-7afe-4caf-a751-698869e323a3 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:01.144062 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:01.188778 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a327d36f-5bf1-4355-ba01-b653339d2118 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:01.189671 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a327d36f-5bf1-4355-ba01-b653339d2118 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:01.189790 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a327d36f-5bf1-4355-ba01-b653339d2118 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:01.190066 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a327d36f-5bf1-4355-ba01-b653339d2118 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:01.206416 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8248ccb-ddd0-4166-ba9c-716d700d1e43 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:01.215803 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a327d36f-5bf1-4355-ba01-b653339d2118 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:59:01.219108 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:01.222050 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-a327d36f-5bf1-4355-ba01-b653339d2118 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 30 04:59:01.222050 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:01.222050 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:01.222050 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:01.222050 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:01.222050 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:01.222050 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:01.222050 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:01.222050 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:01.222050 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:01.222050 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:01.222050 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:01.222050 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:59:01.222050 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:59:01.222050 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:59:01.222050 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:59:01.223690 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:59:01.223690 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:59:01.223690 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:59:01.223690 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:59:01.223690 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:01.223690 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:01.223690 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:01.223690 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:01.223690 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:01.223690 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:01.223690 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:01.223690 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:01.223690 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:01.223690 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:01.223690 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:01.223690 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:01.224911 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:01.224911 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:01.224911 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:01.224911 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:01.224911 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:59:01.224911 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:59:01.224911 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:59:01.224911 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:59:01.224911 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:59:01.224911 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:59:01.224911 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 30 04:59:01.224911 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 04:59:01.228951 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 132/258] 10.222.0.77 () {56 vars in 1114 bytes} [Tue Aug 30 04:59:00 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 240 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 04:59:01.230092 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a327d36f-5bf1-4355-ba01-b653339d2118 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:01.256635 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a93a6333-32f8-433f-a274-6f69b8c32bf6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:01.257603 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a93a6333-32f8-433f-a274-6f69b8c32bf6 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:01.258028 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a93a6333-32f8-433f-a274-6f69b8c32bf6 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:01.258634 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a93a6333-32f8-433f-a274-6f69b8c32bf6 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:01.273148 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a93a6333-32f8-433f-a274-6f69b8c32bf6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:59:01.279646 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-a93a6333-32f8-433f-a274-6f69b8c32bf6 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 04:59:01.279646 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:01.279646 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:01.279646 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:01.279646 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:01.279646 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:01.279646 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:01.279646 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:01.279646 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:01.279646 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:01.279646 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:01.279646 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:01.279646 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:59:01.279646 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:59:01.279646 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:59:01.279646 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:59:01.280697 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:59:01.280697 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:59:01.280697 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:59:01.280697 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:59:01.280697 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:01.280697 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:01.280697 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:01.280697 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:01.280697 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:01.280697 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:01.280697 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:01.280697 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:01.280697 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:01.280697 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:01.280697 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:01.280697 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:01.281765 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:01.281765 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:01.281765 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:01.281765 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:01.281765 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:59:01.281765 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:59:01.281765 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:59:01.281765 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:59:01.281765 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:59:01.281765 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:59:01.281765 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 04:59:01.281765 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 04:59:01.283605 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 127/259] 10.222.0.77 () {56 vars in 1096 bytes} [Tue Aug 30 04:59:01 2022] GET /identity/v3/roles/admin => generated 83 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 04:59:01.289350 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a93a6333-32f8-433f-a274-6f69b8c32bf6 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:01.420711 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a327d36f-5bf1-4355-ba01-b653339d2118 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:01.449881 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a93a6333-32f8-433f-a274-6f69b8c32bf6 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:01.477411 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aec7f0ea-a3a3-4079-b65e-8fcb2adb327e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:01.478025 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aec7f0ea-a3a3-4079-b65e-8fcb2adb327e None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:01.478304 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aec7f0ea-a3a3-4079-b65e-8fcb2adb327e None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:01.478844 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aec7f0ea-a3a3-4079-b65e-8fcb2adb327e None admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:01.479517 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aec7f0ea-a3a3-4079-b65e-8fcb2adb327e None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:01.479820 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aec7f0ea-a3a3-4079-b65e-8fcb2adb327e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:01.481203 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aec7f0ea-a3a3-4079-b65e-8fcb2adb327e None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:01.493653 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0e00faa-3913-4b60-8897-7bd324574799 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:01.494266 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0e00faa-3913-4b60-8897-7bd324574799 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:01.494544 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0e00faa-3913-4b60-8897-7bd324574799 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:01.494811 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0e00faa-3913-4b60-8897-7bd324574799 None admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:01.495499 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0e00faa-3913-4b60-8897-7bd324574799 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:01.495799 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0e00faa-3913-4b60-8897-7bd324574799 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:01.497172 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0e00faa-3913-4b60-8897-7bd324574799 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:01.502065 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 133/260] 10.222.0.77 () {56 vars in 1111 bytes} [Tue Aug 30 04:59:01 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 272 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:01.507568 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aec7f0ea-a3a3-4079-b65e-8fcb2adb327e None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:01.518563 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 128/261] 10.222.0.77 () {56 vars in 1099 bytes} [Tue Aug 30 04:59:01 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:59:01.523723 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0e00faa-3913-4b60-8897-7bd324574799 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:01.543712 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:01.585674 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:01.687019 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aec7f0ea-a3a3-4079-b65e-8fcb2adb327e None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:01.698532 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0e00faa-3913-4b60-8897-7bd324574799 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:01.754278 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5727f4a-5b15-4e47-8138-a86dde2b1cd2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:01.754956 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5727f4a-5b15-4e47-8138-a86dde2b1cd2 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:01.755238 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5727f4a-5b15-4e47-8138-a86dde2b1cd2 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:01.755512 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5727f4a-5b15-4e47-8138-a86dde2b1cd2 None admin] PATH_INFO: `/v3/users/212278f6393840e1848128c611251d64` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:01.759697 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5727f4a-5b15-4e47-8138-a86dde2b1cd2 None admin] RBAC: Authorizing `identity:get_user(user_id=212278f6393840e1848128c611251d64)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:01.761157 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5727f4a-5b15-4e47-8138-a86dde2b1cd2 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:01.762763 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 129/262] 10.222.0.77 () {56 vars in 1177 bytes} [Tue Aug 30 04:59:01 2022] GET /identity/v3/users/212278f6393840e1848128c611251d64 => generated 283 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:59:01.768156 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9254ad84-c6d5-4519-a644-ea7e894244f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:01.768297 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5727f4a-5b15-4e47-8138-a86dde2b1cd2 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:01.768802 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9254ad84-c6d5-4519-a644-ea7e894244f5 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:01.769081 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9254ad84-c6d5-4519-a644-ea7e894244f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:01.769354 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9254ad84-c6d5-4519-a644-ea7e894244f5 None admin] PATH_INFO: `/v3/users/212278f6393840e1848128c611251d64` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:01.772120 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9254ad84-c6d5-4519-a644-ea7e894244f5 None admin] RBAC: Authorizing `identity:get_user(user_id=212278f6393840e1848128c611251d64)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:01.773711 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9254ad84-c6d5-4519-a644-ea7e894244f5 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:01.776296 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 134/263] 10.222.0.77 () {56 vars in 1177 bytes} [Tue Aug 30 04:59:01 2022] GET /identity/v3/users/212278f6393840e1848128c611251d64 => generated 283 bytes in 269 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:01.796762 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9254ad84-c6d5-4519-a644-ea7e894244f5 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:01.876172 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:01.886090 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5727f4a-5b15-4e47-8138-a86dde2b1cd2 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:01.943189 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d3e015d-c3c1-4c1a-adc7-842c797b0ac7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:01.943718 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d3e015d-c3c1-4c1a-adc7-842c797b0ac7 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:01.943996 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d3e015d-c3c1-4c1a-adc7-842c797b0ac7 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:01.944263 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d3e015d-c3c1-4c1a-adc7-842c797b0ac7 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:01.968034 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:01.973030 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9254ad84-c6d5-4519-a644-ea7e894244f5 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:02.042345 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78cdcbd4-f370-4068-982d-dd603514c388 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:02.042997 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78cdcbd4-f370-4068-982d-dd603514c388 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:02.043276 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78cdcbd4-f370-4068-982d-dd603514c388 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:02.043546 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78cdcbd4-f370-4068-982d-dd603514c388 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:02.072004 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d3e015d-c3c1-4c1a-adc7-842c797b0ac7 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:02.073479 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d3e015d-c3c1-4c1a-adc7-842c797b0ac7 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:02.179400 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:02.185275 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 130/264] 10.222.0.77 () {58 vars in 1282 bytes} [Tue Aug 30 04:59:01 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 417 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Aug 30 04:59:02.191852 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d3e015d-c3c1-4c1a-adc7-842c797b0ac7 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:02.207969 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78cdcbd4-f370-4068-982d-dd603514c388 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:02.209383 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78cdcbd4-f370-4068-982d-dd603514c388 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:02.300395 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d3e015d-c3c1-4c1a-adc7-842c797b0ac7 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:02.337931 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1133cb6-5b73-4d22-95e1-a2b03689e525 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:02.338851 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1133cb6-5b73-4d22-95e1-a2b03689e525 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:02.339274 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1133cb6-5b73-4d22-95e1-a2b03689e525 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:02.339687 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1133cb6-5b73-4d22-95e1-a2b03689e525 None admin] PATH_INFO: `/v3/projects/a82de84c774f4bfb8eda1692f1d46a24` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:02.340979 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1133cb6-5b73-4d22-95e1-a2b03689e525 None admin] RBAC: Authorizing `identity:get_project(project_id=a82de84c774f4bfb8eda1692f1d46a24)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:02.342625 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1133cb6-5b73-4d22-95e1-a2b03689e525 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:02.344599 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 131/265] 10.222.0.77 () {56 vars in 1186 bytes} [Tue Aug 30 04:59:02 2022] GET /identity/v3/projects/a82de84c774f4bfb8eda1692f1d46a24 => generated 302 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:02.350290 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1133cb6-5b73-4d22-95e1-a2b03689e525 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:02.433672 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:02.438301 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 135/266] 10.222.0.77 () {58 vars in 1282 bytes} [Tue Aug 30 04:59:01 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 642 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 04:59:02.443724 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78cdcbd4-f370-4068-982d-dd603514c388 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:02.482718 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:02.547996 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1133cb6-5b73-4d22-95e1-a2b03689e525 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:02.591435 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ed26013-b069-4b46-b9cc-37d98b2d90f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:02.592039 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ed26013-b069-4b46-b9cc-37d98b2d90f7 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:02.592322 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ed26013-b069-4b46-b9cc-37d98b2d90f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:02.592594 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ed26013-b069-4b46-b9cc-37d98b2d90f7 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:02.593321 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ed26013-b069-4b46-b9cc-37d98b2d90f7 None admin] RBAC: Adding query filter params (role.id=9d9d5d2c270146488249b7ed943280db, scope.project.id=a82de84c774f4bfb8eda1692f1d46a24, user.id=212278f6393840e1848128c611251d64) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:02.593627 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ed26013-b069-4b46-b9cc-37d98b2d90f7 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:02.595241 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ed26013-b069-4b46-b9cc-37d98b2d90f7 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:02.603016 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 132/267] 10.222.0.77 () {56 vars in 1374 bytes} [Tue Aug 30 04:59:02 2022] GET /identity/v3/role_assignments?user.id=212278f6393840e1848128c611251d64&scope.project.id=a82de84c774f4bfb8eda1692f1d46a24&role.id=9d9d5d2c270146488249b7ed943280db => generated 608 bytes in 253 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:59:02.660852 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78cdcbd4-f370-4068-982d-dd603514c388 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:02.704745 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85d83639-cc55-4f82-bcee-34dcaed26cb4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:02.705748 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85d83639-cc55-4f82-bcee-34dcaed26cb4 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:02.706226 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85d83639-cc55-4f82-bcee-34dcaed26cb4 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:02.706736 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85d83639-cc55-4f82-bcee-34dcaed26cb4 None admin] PATH_INFO: `/v3/projects/a82de84c774f4bfb8eda1692f1d46a24` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:02.708186 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85d83639-cc55-4f82-bcee-34dcaed26cb4 None admin] RBAC: Authorizing `identity:get_project(project_id=a82de84c774f4bfb8eda1692f1d46a24)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:02.709884 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85d83639-cc55-4f82-bcee-34dcaed26cb4 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:02.712406 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 136/268] 10.222.0.77 () {56 vars in 1186 bytes} [Tue Aug 30 04:59:02 2022] GET /identity/v3/projects/a82de84c774f4bfb8eda1692f1d46a24 => generated 302 bytes in 269 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:02.718957 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ed26013-b069-4b46-b9cc-37d98b2d90f7 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:02.742998 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e63bb6cb-1bb3-45d9-af07-d67d9dc4732b None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:02.743645 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e63bb6cb-1bb3-45d9-af07-d67d9dc4732b None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:02.744637 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e63bb6cb-1bb3-45d9-af07-d67d9dc4732b None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:02.747119 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 137/269] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:59:02 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:59:02.754023 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a488e48-bf99-421f-8070-40a3f6458ccf None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:02.754635 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a488e48-bf99-421f-8070-40a3f6458ccf None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:02.755148 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a488e48-bf99-421f-8070-40a3f6458ccf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:02.778115 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-0a488e48-bf99-421f-8070-40a3f6458ccf None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:59:02.780125 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-0a488e48-bf99-421f-8070-40a3f6458ccf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:59:02.904484 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:02.912436 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ed26013-b069-4b46-b9cc-37d98b2d90f7 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:02.916944 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 138/270] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:59:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 30 04:59:02.967843 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:02.973617 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28fb9fa8-1569-4a9e-859e-4a24b72fafdc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:02.974542 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28fb9fa8-1569-4a9e-859e-4a24b72fafdc None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:02.975216 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28fb9fa8-1569-4a9e-859e-4a24b72fafdc None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:02.975770 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28fb9fa8-1569-4a9e-859e-4a24b72fafdc None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:02.976876 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28fb9fa8-1569-4a9e-859e-4a24b72fafdc None admin] RBAC: Adding query filter params (role.id=3b9d5509a2f7478c9b4989d921daef2a, scope.project.id=a82de84c774f4bfb8eda1692f1d46a24, user.id=212278f6393840e1848128c611251d64) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:02.977227 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28fb9fa8-1569-4a9e-859e-4a24b72fafdc None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:02.978946 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28fb9fa8-1569-4a9e-859e-4a24b72fafdc None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:02.991392 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 133/271] 10.222.0.77 () {56 vars in 1374 bytes} [Tue Aug 30 04:59:02 2022] GET /identity/v3/role_assignments?user.id=212278f6393840e1848128c611251d64&scope.project.id=a82de84c774f4bfb8eda1692f1d46a24&role.id=3b9d5509a2f7478c9b4989d921daef2a => generated 608 bytes in 273 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:03.235384 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a488e48-bf99-421f-8070-40a3f6458ccf None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:03.240465 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2f9a5cf-b066-49de-955b-95c6a3c9cccb None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:03.245576 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2f9a5cf-b066-49de-955b-95c6a3c9cccb None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:03.246122 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2f9a5cf-b066-49de-955b-95c6a3c9cccb None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:03.247826 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 134/272] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:59:03 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:59:03.255398 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5d110bb-ee58-4f6a-b1d2-172bfbe7f7fb None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:03.255958 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5d110bb-ee58-4f6a-b1d2-172bfbe7f7fb None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:03.256449 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5d110bb-ee58-4f6a-b1d2-172bfbe7f7fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:03.278147 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-f5d110bb-ee58-4f6a-b1d2-172bfbe7f7fb None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:59:03.279759 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-f5d110bb-ee58-4f6a-b1d2-172bfbe7f7fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:59:03.359436 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 135/273] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:59:03 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:59:03.361028 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ef22e4c-095e-4ec0-b476-77a404a60259 None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:03.361538 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ef22e4c-095e-4ec0-b476-77a404a60259 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:03.362006 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ef22e4c-095e-4ec0-b476-77a404a60259 None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:03.363488 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 136/274] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:59:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:59:03.364719 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a488e48-bf99-421f-8070-40a3f6458ccf None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:03.370313 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-941e3532-3217-4006-92b8-0ab4cd339d57 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:03.370601 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-941e3532-3217-4006-92b8-0ab4cd339d57 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:03.370927 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-941e3532-3217-4006-92b8-0ab4cd339d57 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:03.404710 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0ea0fc0-93b9-45c5-893b-d1dbc2cd7f1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:03.405592 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0ea0fc0-93b9-45c5-893b-d1dbc2cd7f1b None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:03.405875 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0ea0fc0-93b9-45c5-893b-d1dbc2cd7f1b None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:03.406148 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0ea0fc0-93b9-45c5-893b-d1dbc2cd7f1b None admin] PATH_INFO: `/v3/roles/member` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:03.411837 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-941e3532-3217-4006-92b8-0ab4cd339d57 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:59:03.413044 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-941e3532-3217-4006-92b8-0ab4cd339d57 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:59:03.417782 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0ea0fc0-93b9-45c5-893b-d1dbc2cd7f1b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:59:03.422739 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-d0ea0fc0-93b9-45c5-893b-d1dbc2cd7f1b None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 30 04:59:03.422739 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:03.422739 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:03.422739 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:03.422739 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:03.422739 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:03.422739 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:03.422739 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:03.422739 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:03.422739 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:03.422739 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:03.422739 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:03.422739 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:59:03.422739 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:59:03.422739 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:59:03.422739 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:59:03.423624 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:59:03.423624 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:59:03.423624 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:59:03.423624 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:59:03.423624 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:03.423624 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:03.423624 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:03.423624 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:03.423624 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:03.423624 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:03.423624 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:03.423624 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:03.423624 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:03.423624 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:03.423624 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:03.423624 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:03.424501 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:03.424501 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:03.424501 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:03.424501 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:03.424501 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:59:03.424501 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:59:03.424501 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:59:03.424501 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:59:03.424501 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:59:03.424501 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:59:03.424501 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 30 04:59:03.424501 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 04:59:03.424501 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 139/275] 10.222.0.77 () {56 vars in 1099 bytes} [Tue Aug 30 04:59:03 2022] GET /identity/v3/roles/member => generated 84 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:03.429573 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0ea0fc0-93b9-45c5-893b-d1dbc2cd7f1b None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:03.502981 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:03.527087 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 137/276] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:59:03 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 30 04:59:03.554301 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0ea0fc0-93b9-45c5-893b-d1dbc2cd7f1b None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:03.590760 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b353c0c-0c41-421c-84be-a4eb10f66258 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:03.591739 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b353c0c-0c41-421c-84be-a4eb10f66258 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:03.592219 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b353c0c-0c41-421c-84be-a4eb10f66258 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:03.592707 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b353c0c-0c41-421c-84be-a4eb10f66258 None admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:03.593598 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b353c0c-0c41-421c-84be-a4eb10f66258 None admin] RBAC: Adding query filter params (name=member) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:03.594144 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b353c0c-0c41-421c-84be-a4eb10f66258 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:03.595761 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b353c0c-0c41-421c-84be-a4eb10f66258 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:03.605431 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 140/277] 10.222.0.77 () {56 vars in 1101 bytes} [Tue Aug 30 04:59:03 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:03.615416 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-941e3532-3217-4006-92b8-0ab4cd339d57 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:03.667439 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b353c0c-0c41-421c-84be-a4eb10f66258 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:03.675687 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:03.724626 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-941e3532-3217-4006-92b8-0ab4cd339d57 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:03.761095 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8d124f7-642f-49cc-8ac3-1cf9097b9ad4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:03.761635 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d124f7-642f-49cc-8ac3-1cf9097b9ad4 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:03.761908 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d124f7-642f-49cc-8ac3-1cf9097b9ad4 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:03.762174 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d124f7-642f-49cc-8ac3-1cf9097b9ad4 None admin] PATH_INFO: `/v3/users/2770a3346d164489b5442cff9c1085b7` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:03.776545 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8d124f7-642f-49cc-8ac3-1cf9097b9ad4 None admin] RBAC: Authorizing `identity:get_user(user_id=2770a3346d164489b5442cff9c1085b7)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:03.778015 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8d124f7-642f-49cc-8ac3-1cf9097b9ad4 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:03.779811 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 138/278] 10.222.0.77 () {56 vars in 1177 bytes} [Tue Aug 30 04:59:03 2022] GET /identity/v3/users/2770a3346d164489b5442cff9c1085b7 => generated 293 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:03.780772 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-772c12cd-f76d-4048-98fb-c729d9bc30d1 None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:03.781046 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-772c12cd-f76d-4048-98fb-c729d9bc30d1 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:03.781311 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-772c12cd-f76d-4048-98fb-c729d9bc30d1 None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:03.782849 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 139/279] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:59:03 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:59:03.787646 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-772c12cd-f76d-4048-98fb-c729d9bc30d1 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:03.796231 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b353c0c-0c41-421c-84be-a4eb10f66258 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:03.817450 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:03.834110 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-536f6242-cef6-4e4c-a64e-e7e5bfa0bd29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:03.834742 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-536f6242-cef6-4e4c-a64e-e7e5bfa0bd29 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:03.835070 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-536f6242-cef6-4e4c-a64e-e7e5bfa0bd29 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:03.835331 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-536f6242-cef6-4e4c-a64e-e7e5bfa0bd29 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:03.836443 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-536f6242-cef6-4e4c-a64e-e7e5bfa0bd29 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:03.838445 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-536f6242-cef6-4e4c-a64e-e7e5bfa0bd29 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:03.840061 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 141/280] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 04:59:03 2022] GET /identity/v3/domains/default => generated 199 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:59:03.841275 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec637e62-31f0-43e0-9d17-5437681df77e None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:03.841568 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec637e62-31f0-43e0-9d17-5437681df77e None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:03.841826 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec637e62-31f0-43e0-9d17-5437681df77e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:03.859277 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-ec637e62-31f0-43e0-9d17-5437681df77e None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:59:03.860460 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-ec637e62-31f0-43e0-9d17-5437681df77e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:59:03.898770 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-772c12cd-f76d-4048-98fb-c729d9bc30d1 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:03.913236 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:03.928746 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 142/281] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:59:03 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:59:03.929912 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec637e62-31f0-43e0-9d17-5437681df77e None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:03.942895 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11794ba1-053f-451d-9c3a-2a9fa3230852 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:03.943610 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11794ba1-053f-451d-9c3a-2a9fa3230852 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:03.943884 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11794ba1-053f-451d-9c3a-2a9fa3230852 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:03.944257 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11794ba1-053f-451d-9c3a-2a9fa3230852 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:03.944971 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11794ba1-053f-451d-9c3a-2a9fa3230852 None admin] RBAC: Adding query filter params (role.id=6f03d86980c34c36b44a8d6764ff4fda, user.id=2770a3346d164489b5442cff9c1085b7, scope.system=all) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:03.945279 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11794ba1-053f-451d-9c3a-2a9fa3230852 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:03.947380 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11794ba1-053f-451d-9c3a-2a9fa3230852 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:03.958888 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 140/282] 10.222.0.77 () {56 vars in 1308 bytes} [Tue Aug 30 04:59:03 2022] GET /identity/v3/role_assignments?user.id=2770a3346d164489b5442cff9c1085b7&scope.system=all&role.id=6f03d86980c34c36b44a8d6764ff4fda => generated 226 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:59:03.958888 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11794ba1-053f-451d-9c3a-2a9fa3230852 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:04.059497 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec637e62-31f0-43e0-9d17-5437681df77e None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:04.086356 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:04.097940 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11794ba1-053f-451d-9c3a-2a9fa3230852 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:04.105737 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd0ea8cc-7b92-486e-a015-7ad2d86f9e36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:04.106730 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd0ea8cc-7b92-486e-a015-7ad2d86f9e36 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:04.107184 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd0ea8cc-7b92-486e-a015-7ad2d86f9e36 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:04.107734 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd0ea8cc-7b92-486e-a015-7ad2d86f9e36 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:04.122990 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd0ea8cc-7b92-486e-a015-7ad2d86f9e36 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:59:04.130218 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-cd0ea8cc-7b92-486e-a015-7ad2d86f9e36 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 30 04:59:04.130218 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:04.130218 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:04.130218 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:04.130218 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:04.130218 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:04.130218 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:04.130218 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:04.130218 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:04.130218 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:04.130218 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:04.130218 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:04.130218 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:59:04.130218 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:59:04.130218 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:59:04.130218 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:59:04.131126 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:59:04.131126 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:59:04.131126 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:59:04.131126 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:59:04.131126 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:04.131126 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:04.131126 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:04.131126 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:04.131126 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:04.131126 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:04.131126 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:04.131126 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:04.131126 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:04.131126 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:04.131126 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:04.131126 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:04.131905 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:04.131905 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:04.131905 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:04.131905 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:04.131905 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:59:04.131905 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:59:04.131905 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:59:04.131905 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:59:04.131905 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:59:04.131905 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:59:04.131905 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 30 04:59:04.131905 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 04:59:04.132523 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 143/283] 10.222.0.77 () {56 vars in 1099 bytes} [Tue Aug 30 04:59:03 2022] GET /identity/v3/roles/reader => generated 84 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:04.139585 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd0ea8cc-7b92-486e-a015-7ad2d86f9e36 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:04.147192 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95232766-cfae-4f80-b707-999607ab83cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:04.147804 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95232766-cfae-4f80-b707-999607ab83cb None admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:04.148081 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95232766-cfae-4f80-b707-999607ab83cb None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:04.148351 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95232766-cfae-4f80-b707-999607ab83cb None admin] PATH_INFO: `/v3/users` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:04.149119 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95232766-cfae-4f80-b707-999607ab83cb None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:04.150554 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95232766-cfae-4f80-b707-999607ab83cb None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:04.175548 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 141/284] 10.222.0.77 () {60 vars in 1132 bytes} [Tue Aug 30 04:59:03 2022] POST /identity/v3/users => generated 293 bytes in 222 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:59:04.210624 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95232766-cfae-4f80-b707-999607ab83cb None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:04.263486 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:04.275483 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd0ea8cc-7b92-486e-a015-7ad2d86f9e36 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:04.315786 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4337248a-4dc0-473b-883a-5ff942e006e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:04.316321 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4337248a-4dc0-473b-883a-5ff942e006e0 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:04.316604 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4337248a-4dc0-473b-883a-5ff942e006e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:04.316879 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4337248a-4dc0-473b-883a-5ff942e006e0 None admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:04.317567 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4337248a-4dc0-473b-883a-5ff942e006e0 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:04.317846 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4337248a-4dc0-473b-883a-5ff942e006e0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:04.319654 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4337248a-4dc0-473b-883a-5ff942e006e0 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:04.329909 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 144/285] 10.222.0.77 () {56 vars in 1101 bytes} [Tue Aug 30 04:59:04 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:59:04.335817 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4337248a-4dc0-473b-883a-5ff942e006e0 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:04.361195 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95232766-cfae-4f80-b707-999607ab83cb None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:04.386528 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:04.407492 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3b8f017-b520-40fc-98cf-e36129100bf2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:04.408486 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b8f017-b520-40fc-98cf-e36129100bf2 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:04.408988 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b8f017-b520-40fc-98cf-e36129100bf2 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:04.409453 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b8f017-b520-40fc-98cf-e36129100bf2 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:04.422355 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3b8f017-b520-40fc-98cf-e36129100bf2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:59:04.428336 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-a3b8f017-b520-40fc-98cf-e36129100bf2 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 30 04:59:04.428336 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:04.428336 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:04.428336 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:04.428336 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:04.428336 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:04.428336 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:04.428336 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:04.428336 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:04.428336 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:04.428336 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:04.428336 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:04.428336 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:59:04.428336 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:59:04.428336 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:59:04.428336 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:59:04.429360 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:59:04.429360 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:59:04.429360 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:59:04.429360 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:59:04.429360 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:04.429360 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:04.429360 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:04.429360 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:04.429360 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:04.429360 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:04.429360 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:04.429360 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:04.429360 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:04.429360 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:04.429360 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:04.429360 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:04.430398 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:04.430398 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:04.430398 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:04.430398 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:04.430398 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:59:04.430398 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:59:04.430398 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:59:04.430398 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:59:04.430398 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:59:04.430398 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:59:04.430398 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 30 04:59:04.430398 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 04:59:04.432534 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 142/286] 10.222.0.77 () {56 vars in 1099 bytes} [Tue Aug 30 04:59:04 2022] GET /identity/v3/roles/member => generated 84 bytes in 222 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 04:59:04.438384 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3b8f017-b520-40fc-98cf-e36129100bf2 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:04.455455 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4337248a-4dc0-473b-883a-5ff942e006e0 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:04.497545 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0b82291-b0ce-4dbc-821d-732f5807e696 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:04.498290 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0b82291-b0ce-4dbc-821d-732f5807e696 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:04.498666 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0b82291-b0ce-4dbc-821d-732f5807e696 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:04.499023 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0b82291-b0ce-4dbc-821d-732f5807e696 None admin] PATH_INFO: `/v3/users/9f1b91507f9243b5bcd11759ab3a4592` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:04.503185 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0b82291-b0ce-4dbc-821d-732f5807e696 None admin] RBAC: Authorizing `identity:get_user(user_id=9f1b91507f9243b5bcd11759ab3a4592)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:04.504655 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0b82291-b0ce-4dbc-821d-732f5807e696 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:04.506262 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 145/287] 10.222.0.77 () {56 vars in 1177 bytes} [Tue Aug 30 04:59:04 2022] GET /identity/v3/users/9f1b91507f9243b5bcd11759ab3a4592 => generated 297 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:04.513045 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0b82291-b0ce-4dbc-821d-732f5807e696 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:04.565447 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:04.575085 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3b8f017-b520-40fc-98cf-e36129100bf2 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:04.612066 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cce9d31-512e-4bc7-8e4f-bf9d08689632 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:04.612677 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cce9d31-512e-4bc7-8e4f-bf9d08689632 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:04.612951 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cce9d31-512e-4bc7-8e4f-bf9d08689632 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:04.613214 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cce9d31-512e-4bc7-8e4f-bf9d08689632 None admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:04.613871 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cce9d31-512e-4bc7-8e4f-bf9d08689632 None admin] RBAC: Adding query filter params (name=member) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:04.614171 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cce9d31-512e-4bc7-8e4f-bf9d08689632 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:04.615760 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cce9d31-512e-4bc7-8e4f-bf9d08689632 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:04.627127 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 143/288] 10.222.0.77 () {56 vars in 1101 bytes} [Tue Aug 30 04:59:04 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:04.632428 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cce9d31-512e-4bc7-8e4f-bf9d08689632 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:04.645970 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0b82291-b0ce-4dbc-821d-732f5807e696 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:04.679212 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:04.686260 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9c6122d-a788-48c0-950b-10bb190a9c52 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:04.689902 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9c6122d-a788-48c0-950b-10bb190a9c52 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:04.689902 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9c6122d-a788-48c0-950b-10bb190a9c52 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:04.689902 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9c6122d-a788-48c0-950b-10bb190a9c52 None admin] PATH_INFO: `/v3/projects/a82de84c774f4bfb8eda1692f1d46a24` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:04.693014 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9c6122d-a788-48c0-950b-10bb190a9c52 None admin] RBAC: Authorizing `identity:get_project(project_id=a82de84c774f4bfb8eda1692f1d46a24)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:04.694537 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9c6122d-a788-48c0-950b-10bb190a9c52 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:04.697780 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 146/289] 10.222.0.77 () {56 vars in 1186 bytes} [Tue Aug 30 04:59:04 2022] GET /identity/v3/projects/a82de84c774f4bfb8eda1692f1d46a24 => generated 302 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:59:04.720587 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9c6122d-a788-48c0-950b-10bb190a9c52 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:04.816856 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cce9d31-512e-4bc7-8e4f-bf9d08689632 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:04.856492 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dd7e87b-b65a-4970-b3f1-ba1a5998b384 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:04.857176 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dd7e87b-b65a-4970-b3f1-ba1a5998b384 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:04.857471 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dd7e87b-b65a-4970-b3f1-ba1a5998b384 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:04.857745 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dd7e87b-b65a-4970-b3f1-ba1a5998b384 None admin] PATH_INFO: `/v3/users/f8503162a9e14946a7a7272c543757c9` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:04.858989 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dd7e87b-b65a-4970-b3f1-ba1a5998b384 None admin] RBAC: Authorizing `identity:get_user(user_id=f8503162a9e14946a7a7272c543757c9)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:04.860750 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dd7e87b-b65a-4970-b3f1-ba1a5998b384 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:04.862698 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 144/290] 10.222.0.77 () {56 vars in 1177 bytes} [Tue Aug 30 04:59:04 2022] GET /identity/v3/users/f8503162a9e14946a7a7272c543757c9 => generated 297 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:04.868508 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dd7e87b-b65a-4970-b3f1-ba1a5998b384 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:04.900412 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:04.931835 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9c6122d-a788-48c0-950b-10bb190a9c52 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:04.995063 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dd7e87b-b65a-4970-b3f1-ba1a5998b384 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:05.014856 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87ee2edb-855d-40d6-8724-4dad17f064a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:05.015948 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87ee2edb-855d-40d6-8724-4dad17f064a4 None admin] REQUEST_METHOD: `PUT` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:05.017531 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87ee2edb-855d-40d6-8724-4dad17f064a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:05.017969 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87ee2edb-855d-40d6-8724-4dad17f064a4 None admin] PATH_INFO: `/v3/projects/a82de84c774f4bfb8eda1692f1d46a24/users/9f1b91507f9243b5bcd11759ab3a4592/roles/ddaf2963f700481eb93adea7a36f4e73` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:05.023021 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87ee2edb-855d-40d6-8724-4dad17f064a4 None admin] RBAC: Authorizing `identity:create_grant(project_id=a82de84c774f4bfb8eda1692f1d46a24, user_id=9f1b91507f9243b5bcd11759ab3a4592, role_id=ddaf2963f700481eb93adea7a36f4e73)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:05.024682 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87ee2edb-855d-40d6-8724-4dad17f064a4 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:05.047798 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 147/291] 10.222.0.77 () {58 vars in 1439 bytes} [Tue Aug 30 04:59:04 2022] PUT /identity/v3/projects/a82de84c774f4bfb8eda1692f1d46a24/users/9f1b91507f9243b5bcd11759ab3a4592/roles/ddaf2963f700481eb93adea7a36f4e73 => generated 0 bytes in 328 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 04:59:05.059223 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a1f974b-6130-4908-9942-9d8175a086a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:05.060088 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a1f974b-6130-4908-9942-9d8175a086a7 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:05.060512 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a1f974b-6130-4908-9942-9d8175a086a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:05.060928 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a1f974b-6130-4908-9942-9d8175a086a7 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:05.153890 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a1f974b-6130-4908-9942-9d8175a086a7 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:05.155655 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a1f974b-6130-4908-9942-9d8175a086a7 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:05.190236 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:05.280936 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 145/292] 10.222.0.77 () {58 vars in 1282 bytes} [Tue Aug 30 04:59:04 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 412 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Aug 30 04:59:05.285612 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87ee2edb-855d-40d6-8724-4dad17f064a4 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:05.328861 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:05.391024 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87ee2edb-855d-40d6-8724-4dad17f064a4 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:05.425398 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7092fafb-05d8-41b4-88ac-520a41cb88c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:05.425943 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7092fafb-05d8-41b4-88ac-520a41cb88c3 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:05.426228 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7092fafb-05d8-41b4-88ac-520a41cb88c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:05.426499 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7092fafb-05d8-41b4-88ac-520a41cb88c3 None admin] PATH_INFO: `/v3/projects/a82de84c774f4bfb8eda1692f1d46a24` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:05.427655 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7092fafb-05d8-41b4-88ac-520a41cb88c3 None admin] RBAC: Authorizing `identity:get_project(project_id=a82de84c774f4bfb8eda1692f1d46a24)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:05.429185 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7092fafb-05d8-41b4-88ac-520a41cb88c3 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:05.436324 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 148/293] 10.222.0.77 () {56 vars in 1186 bytes} [Tue Aug 30 04:59:05 2022] GET /identity/v3/projects/a82de84c774f4bfb8eda1692f1d46a24 => generated 302 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:05.437680 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a1f974b-6130-4908-9942-9d8175a086a7 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:05.582570 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a1f974b-6130-4908-9942-9d8175a086a7 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:05.624009 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:05.635997 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dddc5a24-a232-4c08-8248-18634281e7a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:05.636607 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dddc5a24-a232-4c08-8248-18634281e7a1 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:05.636881 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dddc5a24-a232-4c08-8248-18634281e7a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:05.637146 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dddc5a24-a232-4c08-8248-18634281e7a1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:05.637870 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dddc5a24-a232-4c08-8248-18634281e7a1 None admin] RBAC: Adding query filter params (role.id=6f03d86980c34c36b44a8d6764ff4fda, scope.project.id=a82de84c774f4bfb8eda1692f1d46a24, user.id=f8503162a9e14946a7a7272c543757c9) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:05.638168 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dddc5a24-a232-4c08-8248-18634281e7a1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:05.639623 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dddc5a24-a232-4c08-8248-18634281e7a1 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:05.646202 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 146/294] 10.222.0.77 () {56 vars in 1374 bytes} [Tue Aug 30 04:59:05 2022] GET /identity/v3/role_assignments?user.id=f8503162a9e14946a7a7272c543757c9&scope.project.id=a82de84c774f4bfb8eda1692f1d46a24&role.id=6f03d86980c34c36b44a8d6764ff4fda => generated 608 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:06.172288 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecd8bb8b-d31b-45ea-8491-797e5d343441 None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:06.172288 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecd8bb8b-d31b-45ea-8491-797e5d343441 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:06.172288 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecd8bb8b-d31b-45ea-8491-797e5d343441 None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:06.172288 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 149/295] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:59:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:59:06.178077 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f967ef0-b98f-4b3e-8d58-2264edafb404 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:06.178357 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f967ef0-b98f-4b3e-8d58-2264edafb404 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:06.178626 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f967ef0-b98f-4b3e-8d58-2264edafb404 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:06.205990 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-1f967ef0-b98f-4b3e-8d58-2264edafb404 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:59:06.207262 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-1f967ef0-b98f-4b3e-8d58-2264edafb404 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:59:06.291256 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 147/296] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:59:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:59:06.574928 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecd8bb8b-d31b-45ea-8491-797e5d343441 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:06.615564 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f93eda-22d6-4d42-8cf3-a770b1c6d511 None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:06.616071 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f93eda-22d6-4d42-8cf3-a770b1c6d511 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:06.616146 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f93eda-22d6-4d42-8cf3-a770b1c6d511 None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:06.617429 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 148/297] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:59:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:59:06.622858 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4185d5b5-82e4-49e4-a2bb-05c2aa068919 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:06.623143 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4185d5b5-82e4-49e4-a2bb-05c2aa068919 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:06.623413 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4185d5b5-82e4-49e4-a2bb-05c2aa068919 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:06.642265 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-4185d5b5-82e4-49e4-a2bb-05c2aa068919 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:59:06.643474 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-4185d5b5-82e4-49e4-a2bb-05c2aa068919 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:59:06.697955 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecd8bb8b-d31b-45ea-8491-797e5d343441 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:06.701564 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:06.722985 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 149/298] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:59:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 30 04:59:06.746286 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3671e56d-d08b-4bb7-b6bf-96f495d9cacb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:06.747226 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3671e56d-d08b-4bb7-b6bf-96f495d9cacb None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:06.747709 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3671e56d-d08b-4bb7-b6bf-96f495d9cacb None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:06.748155 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3671e56d-d08b-4bb7-b6bf-96f495d9cacb None admin] PATH_INFO: `/v3/roles/member` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:06.759313 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3671e56d-d08b-4bb7-b6bf-96f495d9cacb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:59:06.764607 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-3671e56d-d08b-4bb7-b6bf-96f495d9cacb None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 30 04:59:06.764607 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:06.764607 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:06.764607 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:06.764607 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:06.764607 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:06.764607 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:06.764607 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:06.764607 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:06.764607 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:06.764607 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:06.764607 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:06.764607 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:59:06.764607 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:59:06.764607 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:59:06.764607 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:59:06.765467 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:59:06.765467 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:59:06.765467 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:59:06.765467 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:59:06.765467 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:06.765467 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:06.765467 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:06.765467 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:06.765467 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:06.765467 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:06.765467 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:06.765467 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:06.765467 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:06.765467 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:06.765467 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:06.765467 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:06.766308 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:06.766308 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:06.766308 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:06.766308 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:06.766308 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:59:06.766308 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:59:06.766308 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:59:06.766308 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:59:06.766308 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:59:06.766308 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:59:06.766308 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 30 04:59:06.766308 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 04:59:06.768395 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 150/299] 10.222.0.77 () {56 vars in 1099 bytes} [Tue Aug 30 04:59:06 2022] GET /identity/v3/roles/member => generated 84 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:06.774311 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4185d5b5-82e4-49e4-a2bb-05c2aa068919 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:06.826319 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:06.872003 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df5fa48d-549b-472d-bd34-c98c4f11da04 None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:06.872443 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df5fa48d-549b-472d-bd34-c98c4f11da04 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:06.872825 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df5fa48d-549b-472d-bd34-c98c4f11da04 None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:06.877997 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 151/300] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:59:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 04:59:06.879464 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6099e344-bef4-4837-b246-50520da4f446 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:06.879849 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6099e344-bef4-4837-b246-50520da4f446 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:06.880223 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6099e344-bef4-4837-b246-50520da4f446 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:06.881107 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4185d5b5-82e4-49e4-a2bb-05c2aa068919 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:06.899660 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-6099e344-bef4-4837-b246-50520da4f446 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:59:06.901096 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-6099e344-bef4-4837-b246-50520da4f446 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:59:06.923708 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f715ad68-a1c3-4cb0-a541-4e5dbfa30947 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:06.924536 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f715ad68-a1c3-4cb0-a541-4e5dbfa30947 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:06.925469 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f715ad68-a1c3-4cb0-a541-4e5dbfa30947 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:06.925921 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f715ad68-a1c3-4cb0-a541-4e5dbfa30947 None admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:06.927055 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f715ad68-a1c3-4cb0-a541-4e5dbfa30947 None admin] RBAC: Adding query filter params (name=member) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:06.927502 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f715ad68-a1c3-4cb0-a541-4e5dbfa30947 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:06.929717 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f715ad68-a1c3-4cb0-a541-4e5dbfa30947 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:06.941018 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 150/301] 10.222.0.77 () {56 vars in 1101 bytes} [Tue Aug 30 04:59:06 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:06.946623 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f715ad68-a1c3-4cb0-a541-4e5dbfa30947 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:06.989020 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 152/302] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:59:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 30 04:59:06.994353 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6099e344-bef4-4837-b246-50520da4f446 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:07.055846 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f715ad68-a1c3-4cb0-a541-4e5dbfa30947 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:07.066951 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:07.094874 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0634531-3b51-4923-85ae-771ed121982c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:07.095923 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0634531-3b51-4923-85ae-771ed121982c None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:07.096399 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0634531-3b51-4923-85ae-771ed121982c None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:07.096848 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0634531-3b51-4923-85ae-771ed121982c None admin] PATH_INFO: `/v3/users/2770a3346d164489b5442cff9c1085b7` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:07.098144 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0634531-3b51-4923-85ae-771ed121982c None admin] RBAC: Authorizing `identity:get_user(user_id=2770a3346d164489b5442cff9c1085b7)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:07.099813 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0634531-3b51-4923-85ae-771ed121982c None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:07.101643 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 151/303] 10.222.0.77 () {56 vars in 1177 bytes} [Tue Aug 30 04:59:06 2022] GET /identity/v3/users/2770a3346d164489b5442cff9c1085b7 => generated 293 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:59:07.107752 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6099e344-bef4-4837-b246-50520da4f446 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:07.108457 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0634531-3b51-4923-85ae-771ed121982c None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:07.115185 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:07.143468 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-527fb8d1-af54-4e18-9516-45d4bd35d186 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:07.144111 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-527fb8d1-af54-4e18-9516-45d4bd35d186 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:07.144401 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-527fb8d1-af54-4e18-9516-45d4bd35d186 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:07.144681 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-527fb8d1-af54-4e18-9516-45d4bd35d186 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:07.159591 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-527fb8d1-af54-4e18-9516-45d4bd35d186 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:59:07.166313 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-527fb8d1-af54-4e18-9516-45d4bd35d186 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 30 04:59:07.166313 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:07.166313 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:07.166313 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:07.166313 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:07.166313 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:07.166313 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:07.166313 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:07.166313 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:07.166313 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:07.166313 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:07.166313 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:07.166313 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:59:07.166313 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:59:07.166313 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:59:07.166313 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:59:07.167261 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:59:07.167261 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:59:07.167261 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:59:07.167261 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:59:07.167261 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:07.167261 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:07.167261 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:07.167261 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:07.167261 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:07.167261 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:07.167261 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:07.167261 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:07.167261 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:07.167261 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:07.167261 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:07.167261 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:07.168214 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:07.168214 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:07.168214 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:07.168214 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:07.168214 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:59:07.168214 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:59:07.168214 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:59:07.168214 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:59:07.168214 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:59:07.168214 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:59:07.168214 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 30 04:59:07.168214 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 04:59:07.170862 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 153/304] 10.222.0.77 () {56 vars in 1099 bytes} [Tue Aug 30 04:59:06 2022] GET /identity/v3/roles/reader => generated 84 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 04:59:07.177326 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-527fb8d1-af54-4e18-9516-45d4bd35d186 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:07.231620 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0634531-3b51-4923-85ae-771ed121982c None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:07.266748 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f1363fd-7a8b-413b-a3d7-37bf7dc66072 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:07.267475 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f1363fd-7a8b-413b-a3d7-37bf7dc66072 None admin] REQUEST_METHOD: `PUT` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:07.267860 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f1363fd-7a8b-413b-a3d7-37bf7dc66072 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:07.268236 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f1363fd-7a8b-413b-a3d7-37bf7dc66072 None admin] PATH_INFO: `/v3/system/users/2770a3346d164489b5442cff9c1085b7/roles/6f03d86980c34c36b44a8d6764ff4fda` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:07.269933 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f1363fd-7a8b-413b-a3d7-37bf7dc66072 None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=2770a3346d164489b5442cff9c1085b7, role_id=6f03d86980c34c36b44a8d6764ff4fda)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:07.271572 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f1363fd-7a8b-413b-a3d7-37bf7dc66072 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:07.290307 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 152/305] 10.222.0.77 () {58 vars in 1334 bytes} [Tue Aug 30 04:59:07 2022] PUT /identity/v3/system/users/2770a3346d164489b5442cff9c1085b7/roles/6f03d86980c34c36b44a8d6764ff4fda => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 04:59:07.291642 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f1363fd-7a8b-413b-a3d7-37bf7dc66072 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:07.311105 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-527fb8d1-af54-4e18-9516-45d4bd35d186 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:07.350037 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bb13ae4-5a7b-4628-9bd0-d96631c8855c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:07.350655 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bb13ae4-5a7b-4628-9bd0-d96631c8855c None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:07.350964 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bb13ae4-5a7b-4628-9bd0-d96631c8855c None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:07.351233 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bb13ae4-5a7b-4628-9bd0-d96631c8855c None admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:07.351911 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bb13ae4-5a7b-4628-9bd0-d96631c8855c None admin] RBAC: Adding query filter params (name=reader) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:07.352819 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bb13ae4-5a7b-4628-9bd0-d96631c8855c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:07.354503 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bb13ae4-5a7b-4628-9bd0-d96631c8855c None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:07.356764 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:07.368753 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 154/306] 10.222.0.77 () {56 vars in 1101 bytes} [Tue Aug 30 04:59:07 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:07.370599 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bb13ae4-5a7b-4628-9bd0-d96631c8855c None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:07.392316 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:07.451401 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f1363fd-7a8b-413b-a3d7-37bf7dc66072 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:07.505130 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e50a40b9-43e3-4349-9a21-1bc6bf027546 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:07.505765 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e50a40b9-43e3-4349-9a21-1bc6bf027546 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:07.506048 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e50a40b9-43e3-4349-9a21-1bc6bf027546 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:07.506321 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e50a40b9-43e3-4349-9a21-1bc6bf027546 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:07.507483 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e50a40b9-43e3-4349-9a21-1bc6bf027546 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:07.509458 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e50a40b9-43e3-4349-9a21-1bc6bf027546 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:07.510035 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bb13ae4-5a7b-4628-9bd0-d96631c8855c None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:07.511412 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 153/307] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 04:59:07 2022] GET /identity/v3/domains/default => generated 199 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:07.516519 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e50a40b9-43e3-4349-9a21-1bc6bf027546 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:07.551129 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7bda583-7893-4975-94e7-f7339f4b0ae9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:07.551762 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7bda583-7893-4975-94e7-f7339f4b0ae9 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:07.552054 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7bda583-7893-4975-94e7-f7339f4b0ae9 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:07.552309 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7bda583-7893-4975-94e7-f7339f4b0ae9 None admin] PATH_INFO: `/v3/users/800bfa69da174787ad3469d55da101b4` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:07.566237 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7bda583-7893-4975-94e7-f7339f4b0ae9 None admin] RBAC: Authorizing `identity:get_user(user_id=800bfa69da174787ad3469d55da101b4)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:07.567893 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7bda583-7893-4975-94e7-f7339f4b0ae9 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:07.569810 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 155/308] 10.222.0.77 () {56 vars in 1177 bytes} [Tue Aug 30 04:59:07 2022] GET /identity/v3/users/800bfa69da174787ad3469d55da101b4 => generated 293 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:07.576502 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4078250b-a5a1-4c10-8ce2-96f0fde57c3e None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:07.576786 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4078250b-a5a1-4c10-8ce2-96f0fde57c3e None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:07.577056 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4078250b-a5a1-4c10-8ce2-96f0fde57c3e None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:07.578378 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 156/309] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:59:07 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:59:07.579780 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4078250b-a5a1-4c10-8ce2-96f0fde57c3e None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:07.624991 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e50a40b9-43e3-4349-9a21-1bc6bf027546 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:07.636558 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:07.662454 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:07.664808 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-119606e3-1082-4382-8d2a-5f9599415c9f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:07.665431 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-119606e3-1082-4382-8d2a-5f9599415c9f None admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:07.665716 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-119606e3-1082-4382-8d2a-5f9599415c9f None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:07.665990 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-119606e3-1082-4382-8d2a-5f9599415c9f None admin] PATH_INFO: `/v3/groups` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:07.666884 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-119606e3-1082-4382-8d2a-5f9599415c9f None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:07.668172 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-119606e3-1082-4382-8d2a-5f9599415c9f None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:07.686878 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 154/310] 10.222.0.77 () {60 vars in 1134 bytes} [Tue Aug 30 04:59:07 2022] POST /identity/v3/groups => generated 229 bytes in 169 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:59:07.687042 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b288083b-14b6-4ff3-aa0f-a5d0e8b65d02 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:07.687312 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b288083b-14b6-4ff3-aa0f-a5d0e8b65d02 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:07.687588 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b288083b-14b6-4ff3-aa0f-a5d0e8b65d02 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:07.701733 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4078250b-a5a1-4c10-8ce2-96f0fde57c3e None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:07.714931 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-b288083b-14b6-4ff3-aa0f-a5d0e8b65d02 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:59:07.716576 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-b288083b-14b6-4ff3-aa0f-a5d0e8b65d02 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:59:07.741973 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d56abccf-2c9d-49eb-89a1-1605d91958ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:07.742595 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d56abccf-2c9d-49eb-89a1-1605d91958ed None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:07.742951 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d56abccf-2c9d-49eb-89a1-1605d91958ed None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:07.743221 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d56abccf-2c9d-49eb-89a1-1605d91958ed None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:07.743964 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d56abccf-2c9d-49eb-89a1-1605d91958ed None admin] RBAC: Adding query filter params (role.id=ddaf2963f700481eb93adea7a36f4e73, user.id=800bfa69da174787ad3469d55da101b4, scope.system=all) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:07.744269 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d56abccf-2c9d-49eb-89a1-1605d91958ed None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:07.745886 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d56abccf-2c9d-49eb-89a1-1605d91958ed None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:07.758870 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 157/311] 10.222.0.77 () {56 vars in 1308 bytes} [Tue Aug 30 04:59:07 2022] GET /identity/v3/role_assignments?user.id=800bfa69da174787ad3469d55da101b4&scope.system=all&role.id=ddaf2963f700481eb93adea7a36f4e73 => generated 226 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:07.806350 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 155/312] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:59:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:59:08.075220 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d56abccf-2c9d-49eb-89a1-1605d91958ed None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:08.176419 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d56abccf-2c9d-49eb-89a1-1605d91958ed None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:08.211820 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86fefdb4-f16f-41a6-a72c-bb4a894a940d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:08.212429 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86fefdb4-f16f-41a6-a72c-bb4a894a940d None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:08.212709 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86fefdb4-f16f-41a6-a72c-bb4a894a940d None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:08.212970 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86fefdb4-f16f-41a6-a72c-bb4a894a940d None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:08.221721 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:08.225380 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86fefdb4-f16f-41a6-a72c-bb4a894a940d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:59:08.232440 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-86fefdb4-f16f-41a6-a72c-bb4a894a940d None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 30 04:59:08.232440 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:08.232440 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:08.232440 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:08.232440 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:08.232440 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:08.232440 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:08.232440 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:08.232440 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:08.232440 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:08.232440 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:08.232440 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:08.232440 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:59:08.232440 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:59:08.232440 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:59:08.232440 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:59:08.233332 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:59:08.233332 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:59:08.233332 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:59:08.233332 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:59:08.233332 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:08.233332 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:08.233332 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:08.233332 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:08.233332 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:08.233332 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:08.233332 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:08.233332 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:08.233332 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:08.233332 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:08.233332 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:08.233332 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:08.234399 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:08.234399 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:08.234399 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:08.234399 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:08.234399 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:59:08.234399 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:59:08.234399 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:59:08.234399 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:59:08.234399 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:59:08.234399 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:59:08.234399 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 30 04:59:08.234399 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 04:59:08.234399 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 158/313] 10.222.0.77 () {56 vars in 1099 bytes} [Tue Aug 30 04:59:08 2022] GET /identity/v3/roles/reader => generated 84 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:08.241588 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b288083b-14b6-4ff3-aa0f-a5d0e8b65d02 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:08.344112 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:08.347267 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b288083b-14b6-4ff3-aa0f-a5d0e8b65d02 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:08.384129 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54671353-0a4c-499b-b782-83f9407d83ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:08.384785 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54671353-0a4c-499b-b782-83f9407d83ca None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:08.385059 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54671353-0a4c-499b-b782-83f9407d83ca None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:08.385324 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54671353-0a4c-499b-b782-83f9407d83ca None admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:08.386035 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54671353-0a4c-499b-b782-83f9407d83ca None admin] RBAC: Adding query filter params (name=reader) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:08.386334 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54671353-0a4c-499b-b782-83f9407d83ca None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:08.387737 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54671353-0a4c-499b-b782-83f9407d83ca None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:08.396561 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 156/314] 10.222.0.77 () {56 vars in 1101 bytes} [Tue Aug 30 04:59:08 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:08.404148 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86fefdb4-f16f-41a6-a72c-bb4a894a940d None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:08.524935 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86fefdb4-f16f-41a6-a72c-bb4a894a940d None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:08.559846 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1d3841a-94a4-4ce0-b22f-ce7616bdd5d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:08.560467 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d3841a-94a4-4ce0-b22f-ce7616bdd5d6 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:08.560747 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d3841a-94a4-4ce0-b22f-ce7616bdd5d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:08.561017 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d3841a-94a4-4ce0-b22f-ce7616bdd5d6 None admin] PATH_INFO: `/v3/users/9f1b91507f9243b5bcd11759ab3a4592` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:08.562140 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1d3841a-94a4-4ce0-b22f-ce7616bdd5d6 None admin] RBAC: Authorizing `identity:get_user(user_id=9f1b91507f9243b5bcd11759ab3a4592)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:08.563619 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1d3841a-94a4-4ce0-b22f-ce7616bdd5d6 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:08.565501 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 159/315] 10.222.0.77 () {56 vars in 1177 bytes} [Tue Aug 30 04:59:08 2022] GET /identity/v3/users/9f1b91507f9243b5bcd11759ab3a4592 => generated 297 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:08.571323 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54671353-0a4c-499b-b782-83f9407d83ca None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:08.680533 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54671353-0a4c-499b-b782-83f9407d83ca None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:08.715511 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f107cae-4f72-4bca-9998-3fca467ea69f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:08.716116 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f107cae-4f72-4bca-9998-3fca467ea69f None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:08.716396 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f107cae-4f72-4bca-9998-3fca467ea69f None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:08.716667 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f107cae-4f72-4bca-9998-3fca467ea69f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:08.748601 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:08.788861 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f107cae-4f72-4bca-9998-3fca467ea69f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:08.791023 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f107cae-4f72-4bca-9998-3fca467ea69f None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:08.896219 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 157/316] 10.222.0.77 () {58 vars in 1282 bytes} [Tue Aug 30 04:59:08 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 325 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 04:59:08.904249 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1d3841a-94a4-4ce0-b22f-ce7616bdd5d6 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:08.982955 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:09.017871 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1d3841a-94a4-4ce0-b22f-ce7616bdd5d6 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:09.052901 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e324db4-61ee-43cd-8e57-089a1db354cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:09.053520 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e324db4-61ee-43cd-8e57-089a1db354cd None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:09.053800 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e324db4-61ee-43cd-8e57-089a1db354cd None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:09.054071 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e324db4-61ee-43cd-8e57-089a1db354cd None admin] PATH_INFO: `/v3/projects/a82de84c774f4bfb8eda1692f1d46a24` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:09.055235 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e324db4-61ee-43cd-8e57-089a1db354cd None admin] RBAC: Authorizing `identity:get_project(project_id=a82de84c774f4bfb8eda1692f1d46a24)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:09.056723 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e324db4-61ee-43cd-8e57-089a1db354cd None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:09.058653 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 160/317] 10.222.0.77 () {56 vars in 1186 bytes} [Tue Aug 30 04:59:08 2022] GET /identity/v3/projects/a82de84c774f4bfb8eda1692f1d46a24 => generated 302 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:09.067902 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f107cae-4f72-4bca-9998-3fca467ea69f None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:09.130119 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:09.170849 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f107cae-4f72-4bca-9998-3fca467ea69f None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:09.206018 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e76ef6df-2090-4232-b317-bb1dc2219332 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:09.207033 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e76ef6df-2090-4232-b317-bb1dc2219332 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:09.207511 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e76ef6df-2090-4232-b317-bb1dc2219332 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:09.207994 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e76ef6df-2090-4232-b317-bb1dc2219332 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:09.208911 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e76ef6df-2090-4232-b317-bb1dc2219332 None admin] RBAC: Adding query filter params (role.id=ddaf2963f700481eb93adea7a36f4e73, scope.project.id=a82de84c774f4bfb8eda1692f1d46a24, user.id=9f1b91507f9243b5bcd11759ab3a4592) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:09.209425 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e76ef6df-2090-4232-b317-bb1dc2219332 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:09.211151 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e76ef6df-2090-4232-b317-bb1dc2219332 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:09.222722 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 158/318] 10.222.0.77 () {56 vars in 1374 bytes} [Tue Aug 30 04:59:09 2022] GET /identity/v3/role_assignments?user.id=9f1b91507f9243b5bcd11759ab3a4592&scope.project.id=a82de84c774f4bfb8eda1692f1d46a24&role.id=ddaf2963f700481eb93adea7a36f4e73 => generated 608 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:09.273453 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a2ae45-2ca5-4fba-9590-6d56208eb4bb None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:09.275028 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a2ae45-2ca5-4fba-9590-6d56208eb4bb None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:09.275443 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a2ae45-2ca5-4fba-9590-6d56208eb4bb None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:09.277296 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 161/319] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:59:09 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:59:09.283566 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abdd22a3-0eae-40f3-ba9f-4c9a6c240fcc None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:09.284069 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abdd22a3-0eae-40f3-ba9f-4c9a6c240fcc None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:09.284483 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abdd22a3-0eae-40f3-ba9f-4c9a6c240fcc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:09.304769 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-abdd22a3-0eae-40f3-ba9f-4c9a6c240fcc None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:59:09.306341 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-abdd22a3-0eae-40f3-ba9f-4c9a6c240fcc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:59:09.381153 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 159/320] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:59:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:59:09.645622 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31a2ae45-2ca5-4fba-9590-6d56208eb4bb None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:09.717488 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7efc9a57-1c28-401a-ba94-a3b8601a1f2b None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:09.718146 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7efc9a57-1c28-401a-ba94-a3b8601a1f2b None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:09.718146 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7efc9a57-1c28-401a-ba94-a3b8601a1f2b None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:09.719535 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 160/321] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:59:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:59:09.724370 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb59865e-7a0a-4210-81d6-0cb63635cecb None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:09.724657 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb59865e-7a0a-4210-81d6-0cb63635cecb None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:09.724933 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb59865e-7a0a-4210-81d6-0cb63635cecb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:09.743329 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-bb59865e-7a0a-4210-81d6-0cb63635cecb None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:59:09.744518 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-bb59865e-7a0a-4210-81d6-0cb63635cecb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:59:09.752556 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31a2ae45-2ca5-4fba-9590-6d56208eb4bb None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:09.787952 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4d2063b-af73-4a4a-8b76-e2df4afc12be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:09.788573 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d2063b-af73-4a4a-8b76-e2df4afc12be None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:09.788853 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d2063b-af73-4a4a-8b76-e2df4afc12be None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:09.789120 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d2063b-af73-4a4a-8b76-e2df4afc12be None admin] PATH_INFO: `/v3/roles/member` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:09.791049 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:09.798103 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:09.800630 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4d2063b-af73-4a4a-8b76-e2df4afc12be None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:59:09.805425 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-f4d2063b-af73-4a4a-8b76-e2df4afc12be None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 30 04:59:09.805425 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:09.805425 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:09.805425 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:09.805425 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:09.805425 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:09.805425 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:09.805425 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:09.805425 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:09.805425 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:09.805425 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:09.805425 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:09.805425 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:59:09.805425 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:59:09.805425 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:59:09.805425 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:59:09.806281 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:59:09.806281 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:59:09.806281 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:59:09.806281 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:59:09.806281 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:09.806281 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:09.806281 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:09.806281 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:09.806281 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:09.806281 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:09.806281 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:09.806281 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:09.806281 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:09.806281 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:09.806281 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:09.806281 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:09.807149 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:09.807149 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:09.807149 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:09.807149 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:09.807149 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:59:09.807149 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:59:09.807149 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:59:09.807149 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:59:09.807149 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:59:09.807149 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:59:09.807149 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 30 04:59:09.807149 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 04:59:09.807149 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 162/322] 10.222.0.77 () {56 vars in 1099 bytes} [Tue Aug 30 04:59:09 2022] GET /identity/v3/roles/member => generated 84 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:09.812060 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4d2063b-af73-4a4a-8b76-e2df4afc12be None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:09.818903 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 161/323] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:59:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 30 04:59:09.822713 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac3e9abf-7981-4b5c-bb33-f8b97206c7ef None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:09.823234 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac3e9abf-7981-4b5c-bb33-f8b97206c7ef None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:09.823644 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac3e9abf-7981-4b5c-bb33-f8b97206c7ef None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:09.825312 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 162/324] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:59:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:59:09.829933 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d00c9b77-3073-40ce-b054-83d5bfdb0479 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:09.830911 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d00c9b77-3073-40ce-b054-83d5bfdb0479 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:09.831664 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d00c9b77-3073-40ce-b054-83d5bfdb0479 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:09.854484 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-d00c9b77-3073-40ce-b054-83d5bfdb0479 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:59:09.855985 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-d00c9b77-3073-40ce-b054-83d5bfdb0479 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:59:09.919613 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4d2063b-af73-4a4a-8b76-e2df4afc12be None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:09.935050 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 163/325] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:59:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 30 04:59:09.954268 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8b72f69-e66d-4321-b927-94c1b942167d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:09.954987 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b72f69-e66d-4321-b927-94c1b942167d None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:09.955333 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b72f69-e66d-4321-b927-94c1b942167d None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:09.955671 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b72f69-e66d-4321-b927-94c1b942167d None admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:09.956384 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8b72f69-e66d-4321-b927-94c1b942167d None admin] RBAC: Adding query filter params (name=member) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:09.956770 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8b72f69-e66d-4321-b927-94c1b942167d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:09.958193 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8b72f69-e66d-4321-b927-94c1b942167d None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:09.967263 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 163/326] 10.222.0.77 () {56 vars in 1101 bytes} [Tue Aug 30 04:59:09 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:59:09.972612 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d00c9b77-3073-40ce-b054-83d5bfdb0479 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:10.072084 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d00c9b77-3073-40ce-b054-83d5bfdb0479 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:10.088088 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:10.090694 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8b72f69-e66d-4321-b927-94c1b942167d None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:10.111778 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d919d856-6850-4e31-92a0-94be7e952ad9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:10.112401 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d919d856-6850-4e31-92a0-94be7e952ad9 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:10.113084 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d919d856-6850-4e31-92a0-94be7e952ad9 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:10.113355 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d919d856-6850-4e31-92a0-94be7e952ad9 None admin] PATH_INFO: `/v3/users/2770a3346d164489b5442cff9c1085b7` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:10.114721 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d919d856-6850-4e31-92a0-94be7e952ad9 None admin] RBAC: Authorizing `identity:get_user(user_id=2770a3346d164489b5442cff9c1085b7)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:10.116811 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d919d856-6850-4e31-92a0-94be7e952ad9 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:10.123875 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 164/327] 10.222.0.77 () {56 vars in 1177 bytes} [Tue Aug 30 04:59:09 2022] GET /identity/v3/users/2770a3346d164489b5442cff9c1085b7 => generated 293 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:59:10.125010 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d919d856-6850-4e31-92a0-94be7e952ad9 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:10.151536 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:10.200539 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8b72f69-e66d-4321-b927-94c1b942167d None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:10.227676 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d919d856-6850-4e31-92a0-94be7e952ad9 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:10.233086 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7c8df8c-e456-49c3-a034-aad6889f5155 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:10.238874 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7c8df8c-e456-49c3-a034-aad6889f5155 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:10.238874 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7c8df8c-e456-49c3-a034-aad6889f5155 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:10.238874 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7c8df8c-e456-49c3-a034-aad6889f5155 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:10.238874 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7c8df8c-e456-49c3-a034-aad6889f5155 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:10.238874 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7c8df8c-e456-49c3-a034-aad6889f5155 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:10.238874 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 164/328] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 04:59:10 2022] GET /identity/v3/domains/default => generated 199 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:10.239430 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7c8df8c-e456-49c3-a034-aad6889f5155 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:10.267930 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6db9e343-cb14-468d-a24a-807db2dbff09 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:10.268530 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6db9e343-cb14-468d-a24a-807db2dbff09 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:10.268808 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6db9e343-cb14-468d-a24a-807db2dbff09 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:10.269082 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6db9e343-cb14-468d-a24a-807db2dbff09 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:10.269801 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6db9e343-cb14-468d-a24a-807db2dbff09 None admin] RBAC: Adding query filter params (role.id=6f03d86980c34c36b44a8d6764ff4fda, user.id=2770a3346d164489b5442cff9c1085b7, scope.system=all) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:10.270108 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6db9e343-cb14-468d-a24a-807db2dbff09 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:10.271566 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6db9e343-cb14-468d-a24a-807db2dbff09 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:10.283288 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 165/329] 10.222.0.77 () {56 vars in 1308 bytes} [Tue Aug 30 04:59:10 2022] GET /identity/v3/role_assignments?user.id=2770a3346d164489b5442cff9c1085b7&scope.system=all&role.id=6f03d86980c34c36b44a8d6764ff4fda => generated 510 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:59:10.284072 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6db9e343-cb14-468d-a24a-807db2dbff09 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:10.333660 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:10.364104 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7c8df8c-e456-49c3-a034-aad6889f5155 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:10.393941 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:10.403746 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6db9e343-cb14-468d-a24a-807db2dbff09 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:10.407582 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf4bac11-9f2c-4e90-b60a-1aee7c052a7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:10.408222 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf4bac11-9f2c-4e90-b60a-1aee7c052a7d None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:10.408498 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf4bac11-9f2c-4e90-b60a-1aee7c052a7d None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:10.408767 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf4bac11-9f2c-4e90-b60a-1aee7c052a7d None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:10.421388 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf4bac11-9f2c-4e90-b60a-1aee7c052a7d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:59:10.426727 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-bf4bac11-9f2c-4e90-b60a-1aee7c052a7d None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 30 04:59:10.426727 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:10.426727 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:10.426727 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:10.426727 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:10.426727 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:10.426727 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:10.426727 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:10.426727 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:10.426727 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:10.426727 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:10.426727 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:10.426727 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:59:10.426727 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:59:10.426727 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:59:10.426727 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:59:10.427856 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:59:10.427856 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:59:10.427856 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:59:10.427856 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:59:10.427856 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:10.427856 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:10.427856 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:10.427856 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:10.427856 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:10.427856 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:10.427856 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:10.427856 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:10.427856 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:10.427856 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:10.427856 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:10.427856 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:10.428834 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:10.428834 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:10.428834 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:10.428834 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:10.428834 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:59:10.428834 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:59:10.428834 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:59:10.428834 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:59:10.428834 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:59:10.428834 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:59:10.428834 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 30 04:59:10.428834 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 04:59:10.428834 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 165/330] 10.222.0.77 () {56 vars in 1099 bytes} [Tue Aug 30 04:59:10 2022] GET /identity/v3/roles/reader => generated 84 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:10.435073 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf4bac11-9f2c-4e90-b60a-1aee7c052a7d None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:10.450670 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-798435e6-a7fa-4f1d-aa28-1d3301b8f26e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:10.451249 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-798435e6-a7fa-4f1d-aa28-1d3301b8f26e None admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:10.451532 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-798435e6-a7fa-4f1d-aa28-1d3301b8f26e None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:10.451808 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-798435e6-a7fa-4f1d-aa28-1d3301b8f26e None admin] PATH_INFO: `/v3/groups` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:10.452557 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-798435e6-a7fa-4f1d-aa28-1d3301b8f26e None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:10.453878 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-798435e6-a7fa-4f1d-aa28-1d3301b8f26e None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:10.474084 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 166/331] 10.222.0.77 () {60 vars in 1134 bytes} [Tue Aug 30 04:59:10 2022] POST /identity/v3/groups => generated 226 bytes in 190 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:59:10.553086 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf4bac11-9f2c-4e90-b60a-1aee7c052a7d None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:10.588009 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e6e085c-b3e9-4d0e-a4a5-4248b588a2fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:10.588620 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e6e085c-b3e9-4d0e-a4a5-4248b588a2fe None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:10.588898 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e6e085c-b3e9-4d0e-a4a5-4248b588a2fe None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:10.589168 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e6e085c-b3e9-4d0e-a4a5-4248b588a2fe None admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:10.589835 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e6e085c-b3e9-4d0e-a4a5-4248b588a2fe None admin] RBAC: Adding query filter params (name=reader) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:10.590135 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e6e085c-b3e9-4d0e-a4a5-4248b588a2fe None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:10.591649 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e6e085c-b3e9-4d0e-a4a5-4248b588a2fe None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:10.600425 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 166/332] 10.222.0.77 () {56 vars in 1101 bytes} [Tue Aug 30 04:59:10 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:59:10.606915 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-798435e6-a7fa-4f1d-aa28-1d3301b8f26e None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:10.716467 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-798435e6-a7fa-4f1d-aa28-1d3301b8f26e None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:10.718141 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:10.753945 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dd39325-13c4-4160-8921-23657a9e9137 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:10.754568 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dd39325-13c4-4160-8921-23657a9e9137 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:10.754898 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dd39325-13c4-4160-8921-23657a9e9137 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:10.755141 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dd39325-13c4-4160-8921-23657a9e9137 None admin] PATH_INFO: `/v3/users/800bfa69da174787ad3469d55da101b4` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:10.756261 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dd39325-13c4-4160-8921-23657a9e9137 None admin] RBAC: Authorizing `identity:get_user(user_id=800bfa69da174787ad3469d55da101b4)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:10.757679 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dd39325-13c4-4160-8921-23657a9e9137 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:10.759310 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 167/333] 10.222.0.77 () {56 vars in 1177 bytes} [Tue Aug 30 04:59:10 2022] GET /identity/v3/users/800bfa69da174787ad3469d55da101b4 => generated 293 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:10.764947 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e6e085c-b3e9-4d0e-a4a5-4248b588a2fe None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:10.814898 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:10.873586 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e6e085c-b3e9-4d0e-a4a5-4248b588a2fe None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:10.913792 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c78b77a2-e384-4f0b-b9be-b49c9e258495 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:10.914432 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c78b77a2-e384-4f0b-b9be-b49c9e258495 None admin] REQUEST_METHOD: `PUT` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:10.915443 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c78b77a2-e384-4f0b-b9be-b49c9e258495 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:10.915718 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c78b77a2-e384-4f0b-b9be-b49c9e258495 None admin] PATH_INFO: `/v3/system/users/800bfa69da174787ad3469d55da101b4/roles/ddaf2963f700481eb93adea7a36f4e73` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:10.917753 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c78b77a2-e384-4f0b-b9be-b49c9e258495 None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=800bfa69da174787ad3469d55da101b4, role_id=ddaf2963f700481eb93adea7a36f4e73)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:10.919157 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c78b77a2-e384-4f0b-b9be-b49c9e258495 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:10.930414 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 167/334] 10.222.0.77 () {58 vars in 1334 bytes} [Tue Aug 30 04:59:10 2022] PUT /identity/v3/system/users/800bfa69da174787ad3469d55da101b4/roles/ddaf2963f700481eb93adea7a36f4e73 => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 04:59:12.572303 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-798cd6b8-4cef-4fc3-a795-aa9b16363782 None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:12.572795 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-798cd6b8-4cef-4fc3-a795-aa9b16363782 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:12.572869 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-798cd6b8-4cef-4fc3-a795-aa9b16363782 None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:12.574464 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 168/335] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:59:12 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:59:12.579777 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf698eaa-59a7-4d81-8284-f09b516ae856 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:12.580189 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf698eaa-59a7-4d81-8284-f09b516ae856 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:12.580381 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf698eaa-59a7-4d81-8284-f09b516ae856 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:12.603214 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-cf698eaa-59a7-4d81-8284-f09b516ae856 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:59:12.604677 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-cf698eaa-59a7-4d81-8284-f09b516ae856 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:59:12.685381 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 168/336] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:59:12 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:59:12.834682 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a0f6a3d-432b-41fe-bba3-7755425271ca None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:12.835064 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a0f6a3d-432b-41fe-bba3-7755425271ca None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:12.835262 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a0f6a3d-432b-41fe-bba3-7755425271ca None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:12.836883 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 169/337] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:59:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:59:12.842521 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6001c668-fe19-494e-8df8-f8cbd5ae53b3 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:12.842834 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6001c668-fe19-494e-8df8-f8cbd5ae53b3 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:12.843122 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6001c668-fe19-494e-8df8-f8cbd5ae53b3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:12.864919 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-6001c668-fe19-494e-8df8-f8cbd5ae53b3 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:59:12.866101 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-6001c668-fe19-494e-8df8-f8cbd5ae53b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:59:12.910460 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:12.956891 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 169/338] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:59:12 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 30 04:59:13.303902 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a0f6a3d-432b-41fe-bba3-7755425271ca None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:13.432404 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a0f6a3d-432b-41fe-bba3-7755425271ca None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:13.481489 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b04d693-4579-4d44-8d34-2852bcf28068 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:13.482130 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b04d693-4579-4d44-8d34-2852bcf28068 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:13.482406 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b04d693-4579-4d44-8d34-2852bcf28068 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:13.482675 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b04d693-4579-4d44-8d34-2852bcf28068 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:13.494855 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b04d693-4579-4d44-8d34-2852bcf28068 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:59:13.500333 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-5b04d693-4579-4d44-8d34-2852bcf28068 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 30 04:59:13.500333 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:13.500333 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:13.500333 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:13.500333 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:13.500333 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:13.500333 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:13.500333 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:13.500333 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:13.500333 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:13.500333 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:13.500333 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:13.500333 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:59:13.500333 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:59:13.500333 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:59:13.500333 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:59:13.502697 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:59:13.502697 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:59:13.502697 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:59:13.502697 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:59:13.502697 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:13.502697 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:13.502697 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:13.502697 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:13.502697 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:13.502697 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:13.502697 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:13.502697 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:13.502697 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:13.502697 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:13.502697 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:13.502697 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:13.504662 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:13.504662 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:13.504662 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:13.504662 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:13.504662 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:59:13.504662 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:59:13.504662 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:59:13.504662 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:59:13.504662 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:59:13.504662 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:59:13.504662 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 30 04:59:13.504662 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 04:59:13.504662 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 170/339] 10.222.0.77 () {56 vars in 1099 bytes} [Tue Aug 30 04:59:13 2022] GET /identity/v3/roles/member => generated 84 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 04:59:13.510851 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6001c668-fe19-494e-8df8-f8cbd5ae53b3 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:13.571460 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b04d693-4579-4d44-8d34-2852bcf28068 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:13.620518 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:13.662543 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6001c668-fe19-494e-8df8-f8cbd5ae53b3 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:13.700891 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b04d693-4579-4d44-8d34-2852bcf28068 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:13.716418 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6af90b3-7305-407d-8123-20ba579f775a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:13.717185 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6af90b3-7305-407d-8123-20ba579f775a None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:13.717465 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6af90b3-7305-407d-8123-20ba579f775a None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:13.717733 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6af90b3-7305-407d-8123-20ba579f775a None admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:13.718401 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6af90b3-7305-407d-8123-20ba579f775a None admin] RBAC: Adding query filter params (name=member) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:13.718700 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6af90b3-7305-407d-8123-20ba579f775a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:13.720277 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6af90b3-7305-407d-8123-20ba579f775a None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:13.739604 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 170/340] 10.222.0.77 () {56 vars in 1101 bytes} [Tue Aug 30 04:59:13 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:13.742795 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6af90b3-7305-407d-8123-20ba579f775a None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:13.753846 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f70504f4-6b2f-4eda-b217-cbd233e4b8b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:13.754529 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70504f4-6b2f-4eda-b217-cbd233e4b8b5 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:13.754809 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70504f4-6b2f-4eda-b217-cbd233e4b8b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:13.756183 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70504f4-6b2f-4eda-b217-cbd233e4b8b5 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:13.776979 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f70504f4-6b2f-4eda-b217-cbd233e4b8b5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:59:13.782545 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-f70504f4-6b2f-4eda-b217-cbd233e4b8b5 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 30 04:59:13.782545 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:13.782545 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:13.782545 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:13.782545 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:13.782545 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:13.782545 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:13.782545 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:13.782545 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:13.782545 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:13.782545 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:13.782545 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:13.782545 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:59:13.782545 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:59:13.782545 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:59:13.782545 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:59:13.784681 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:59:13.784681 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:59:13.784681 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:59:13.784681 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:59:13.784681 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:13.784681 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:13.784681 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:13.784681 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:13.784681 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:13.784681 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:13.784681 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:13.784681 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:13.784681 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:13.784681 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:13.784681 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:13.784681 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:13.786630 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:13.786630 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:13.786630 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:13.786630 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:13.786630 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:59:13.786630 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:59:13.786630 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:59:13.786630 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:59:13.786630 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:59:13.786630 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:59:13.786630 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 30 04:59:13.786630 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 04:59:13.788119 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 171/341] 10.222.0.77 () {56 vars in 1114 bytes} [Tue Aug 30 04:59:13 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:13.789549 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f70504f4-6b2f-4eda-b217-cbd233e4b8b5 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:13.827934 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:13.898782 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:13.918432 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f70504f4-6b2f-4eda-b217-cbd233e4b8b5 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:13.941833 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6af90b3-7305-407d-8123-20ba579f775a None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:13.961413 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80479193-58cf-4bec-b938-1d2f4fda7844 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:13.962329 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80479193-58cf-4bec-b938-1d2f4fda7844 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:13.962612 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80479193-58cf-4bec-b938-1d2f4fda7844 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:13.962928 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80479193-58cf-4bec-b938-1d2f4fda7844 None admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:13.963600 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80479193-58cf-4bec-b938-1d2f4fda7844 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:13.963948 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80479193-58cf-4bec-b938-1d2f4fda7844 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:13.965340 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80479193-58cf-4bec-b938-1d2f4fda7844 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:13.977083 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 172/342] 10.222.0.77 () {56 vars in 1111 bytes} [Tue Aug 30 04:59:13 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:59:13.979971 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76887f4a-7423-4dce-910e-1f509cae48e7 None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:13.979971 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76887f4a-7423-4dce-910e-1f509cae48e7 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:13.980164 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76887f4a-7423-4dce-910e-1f509cae48e7 None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:13.985440 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 173/343] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:59:13 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:59:13.985440 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f763f2a6-a058-4e9b-9efa-129e9b4d57f3 None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:13.985440 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f763f2a6-a058-4e9b-9efa-129e9b4d57f3 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:13.985440 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f763f2a6-a058-4e9b-9efa-129e9b4d57f3 None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:13.985440 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 174/344] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:59:13 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:59:13.985440 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f763f2a6-a058-4e9b-9efa-129e9b4d57f3 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:13.989352 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54751a87-3117-43a7-a375-d4875c2a4fc8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:13.990232 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54751a87-3117-43a7-a375-d4875c2a4fc8 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:13.990713 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54751a87-3117-43a7-a375-d4875c2a4fc8 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:13.991212 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54751a87-3117-43a7-a375-d4875c2a4fc8 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:14.076051 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54751a87-3117-43a7-a375-d4875c2a4fc8 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:14.077853 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54751a87-3117-43a7-a375-d4875c2a4fc8 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:14.111077 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:14.124841 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f763f2a6-a058-4e9b-9efa-129e9b4d57f3 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:14.165816 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13317ce7-d51d-4f3f-b932-78dd26cc0eee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:14.166428 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13317ce7-d51d-4f3f-b932-78dd26cc0eee None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:14.166711 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13317ce7-d51d-4f3f-b932-78dd26cc0eee None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:14.167094 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13317ce7-d51d-4f3f-b932-78dd26cc0eee None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:14.194610 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:14.206582 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 171/345] 10.222.0.77 () {58 vars in 1282 bytes} [Tue Aug 30 04:59:13 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 464 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 04:59:14.207992 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3506b8-45f8-45d4-bf8c-9cde53a28c72 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:14.208280 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3506b8-45f8-45d4-bf8c-9cde53a28c72 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:14.208552 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3506b8-45f8-45d4-bf8c-9cde53a28c72 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:14.239239 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-cb3506b8-45f8-45d4-bf8c-9cde53a28c72 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:59:14.240463 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-cb3506b8-45f8-45d4-bf8c-9cde53a28c72 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:59:14.257727 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13317ce7-d51d-4f3f-b932-78dd26cc0eee None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:14.259888 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13317ce7-d51d-4f3f-b932-78dd26cc0eee None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:14.325868 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 172/346] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:59:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:59:14.327239 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58685524-6eaa-437c-8ab6-941e882ad59c None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:14.327525 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58685524-6eaa-437c-8ab6-941e882ad59c None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:14.327797 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58685524-6eaa-437c-8ab6-941e882ad59c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:14.347175 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-58685524-6eaa-437c-8ab6-941e882ad59c None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:59:14.348363 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-58685524-6eaa-437c-8ab6-941e882ad59c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:59:14.399933 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 175/347] 10.222.0.77 () {58 vars in 1282 bytes} [Tue Aug 30 04:59:13 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 414 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 04:59:14.400102 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2c0dfbb-adfc-417a-8e4d-4f7ad0e9183e None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:14.400390 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2c0dfbb-adfc-417a-8e4d-4f7ad0e9183e None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:14.400658 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2c0dfbb-adfc-417a-8e4d-4f7ad0e9183e None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:14.402200 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 176/348] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:59:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:59:14.403551 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2c0dfbb-adfc-417a-8e4d-4f7ad0e9183e None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:14.423698 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:14.439920 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 173/349] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:59:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:59:14.441636 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cc06558-9327-4deb-815f-b73d23a33d9d None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:14.442090 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cc06558-9327-4deb-815f-b73d23a33d9d None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:14.442528 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cc06558-9327-4deb-815f-b73d23a33d9d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:14.464051 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-7cc06558-9327-4deb-815f-b73d23a33d9d None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:59:14.465782 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-7cc06558-9327-4deb-815f-b73d23a33d9d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:59:14.497919 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:14.545516 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 174/350] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:59:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:59:14.546672 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cc06558-9327-4deb-815f-b73d23a33d9d None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:14.621431 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2c0dfbb-adfc-417a-8e4d-4f7ad0e9183e None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:14.660606 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cc06558-9327-4deb-815f-b73d23a33d9d None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:14.687639 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:14.691051 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-455533c6-b083-48b0-81c2-3efe4d519ba0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:14.691681 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-455533c6-b083-48b0-81c2-3efe4d519ba0 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:14.691952 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-455533c6-b083-48b0-81c2-3efe4d519ba0 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:14.692215 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-455533c6-b083-48b0-81c2-3efe4d519ba0 None admin] PATH_INFO: `/v3/projects/9c3b4147cbea42e3a97c54416188a48c` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:14.693376 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-455533c6-b083-48b0-81c2-3efe4d519ba0 None admin] RBAC: Authorizing `identity:get_project(project_id=9c3b4147cbea42e3a97c54416188a48c)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:14.694938 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-455533c6-b083-48b0-81c2-3efe4d519ba0 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:14.700953 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 177/351] 10.222.0.77 () {56 vars in 1186 bytes} [Tue Aug 30 04:59:14 2022] GET /identity/v3/projects/9c3b4147cbea42e3a97c54416188a48c => generated 298 bytes in 294 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:14.700953 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-455533c6-b083-48b0-81c2-3efe4d519ba0 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:14.707491 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11ab5ca4-23e9-4c37-8ed3-65df22ac35b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:14.708330 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ab5ca4-23e9-4c37-8ed3-65df22ac35b5 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:14.708753 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ab5ca4-23e9-4c37-8ed3-65df22ac35b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:14.709164 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ab5ca4-23e9-4c37-8ed3-65df22ac35b5 None admin] PATH_INFO: `/v3/projects/9c3b4147cbea42e3a97c54416188a48c` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:14.710516 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11ab5ca4-23e9-4c37-8ed3-65df22ac35b5 None admin] RBAC: Authorizing `identity:get_project(project_id=9c3b4147cbea42e3a97c54416188a48c)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:14.712682 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11ab5ca4-23e9-4c37-8ed3-65df22ac35b5 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:14.714562 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 175/352] 10.222.0.77 () {56 vars in 1186 bytes} [Tue Aug 30 04:59:14 2022] GET /identity/v3/projects/9c3b4147cbea42e3a97c54416188a48c => generated 298 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:14.715498 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11ab5ca4-23e9-4c37-8ed3-65df22ac35b5 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:14.814082 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-455533c6-b083-48b0-81c2-3efe4d519ba0 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:14.828943 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11ab5ca4-23e9-4c37-8ed3-65df22ac35b5 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:14.835145 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:14.850804 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2fc00a9-2c0a-44df-a05d-dc0b5b7b7948 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:14.851781 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2fc00a9-2c0a-44df-a05d-dc0b5b7b7948 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:14.852625 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2fc00a9-2c0a-44df-a05d-dc0b5b7b7948 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:14.852987 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2fc00a9-2c0a-44df-a05d-dc0b5b7b7948 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:14.864017 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2fc00a9-2c0a-44df-a05d-dc0b5b7b7948 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:59:14.870469 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-e2fc00a9-2c0a-44df-a05d-dc0b5b7b7948 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 30 04:59:14.870469 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:14.870469 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:14.870469 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:14.870469 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:14.870469 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:14.870469 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:14.870469 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:14.870469 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:14.870469 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:14.870469 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:14.870469 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:14.870469 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:59:14.870469 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:59:14.870469 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:59:14.870469 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:59:14.871438 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:59:14.871438 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:59:14.871438 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:59:14.871438 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:59:14.871438 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:14.871438 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:14.871438 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:14.871438 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:14.871438 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:14.871438 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:14.871438 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:14.871438 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:14.871438 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:14.871438 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:14.871438 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:14.871438 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:14.872238 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:14.872238 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:14.872238 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:14.872238 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:14.872238 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:59:14.872238 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:59:14.872238 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:59:14.872238 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:59:14.872238 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:59:14.872238 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:59:14.872238 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 30 04:59:14.872238 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 04:59:14.872238 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 178/353] 10.222.0.77 () {56 vars in 1099 bytes} [Tue Aug 30 04:59:14 2022] GET /identity/v3/roles/reader => generated 84 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:14.873047 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2fc00a9-2c0a-44df-a05d-dc0b5b7b7948 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:14.874483 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c71f85fb-05ff-413a-9359-2aa6bfbb294c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:14.875401 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c71f85fb-05ff-413a-9359-2aa6bfbb294c None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:14.875837 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c71f85fb-05ff-413a-9359-2aa6bfbb294c None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:14.876289 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c71f85fb-05ff-413a-9359-2aa6bfbb294c None admin] PATH_INFO: `/v3/groups/b27079fce8234aa482d7e0497689431c` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:14.883627 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c71f85fb-05ff-413a-9359-2aa6bfbb294c None admin] RBAC: Authorizing `identity:get_group(group_id=b27079fce8234aa482d7e0497689431c)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:14.886383 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c71f85fb-05ff-413a-9359-2aa6bfbb294c None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:14.889257 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 176/354] 10.222.0.77 () {56 vars in 1180 bytes} [Tue Aug 30 04:59:14 2022] GET /identity/v3/groups/b27079fce8234aa482d7e0497689431c => generated 226 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:14.890360 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c71f85fb-05ff-413a-9359-2aa6bfbb294c None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:14.953678 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:14.986981 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2fc00a9-2c0a-44df-a05d-dc0b5b7b7948 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:15.007443 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c71f85fb-05ff-413a-9359-2aa6bfbb294c None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:15.031838 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1f19ec7-41b6-4dbf-996a-82f7acfae00b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:15.032707 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f19ec7-41b6-4dbf-996a-82f7acfae00b None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:15.032814 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f19ec7-41b6-4dbf-996a-82f7acfae00b None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:15.033015 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f19ec7-41b6-4dbf-996a-82f7acfae00b None admin] PATH_INFO: `/v3/groups/b27079fce8234aa482d7e0497689431c` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:15.034873 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1f19ec7-41b6-4dbf-996a-82f7acfae00b None admin] RBAC: Authorizing `identity:get_group(group_id=b27079fce8234aa482d7e0497689431c)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:15.036331 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1f19ec7-41b6-4dbf-996a-82f7acfae00b None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:15.038104 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 179/355] 10.222.0.77 () {56 vars in 1180 bytes} [Tue Aug 30 04:59:14 2022] GET /identity/v3/groups/b27079fce8234aa482d7e0497689431c => generated 226 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:15.038659 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1f19ec7-41b6-4dbf-996a-82f7acfae00b None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:15.046641 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e7d57e6-6351-4a61-84d5-258682a169ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:15.047336 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e7d57e6-6351-4a61-84d5-258682a169ec None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:15.047620 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e7d57e6-6351-4a61-84d5-258682a169ec None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:15.047893 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e7d57e6-6351-4a61-84d5-258682a169ec None admin] PATH_INFO: `/v3/roles/member` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:15.058531 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e7d57e6-6351-4a61-84d5-258682a169ec None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:59:15.063976 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-0e7d57e6-6351-4a61-84d5-258682a169ec None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 30 04:59:15.063976 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:15.063976 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:15.063976 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:15.063976 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:15.063976 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:15.063976 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:15.063976 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:15.063976 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:15.063976 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:15.063976 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:15.063976 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:15.063976 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:59:15.063976 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:59:15.063976 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:59:15.063976 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:59:15.064851 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:59:15.064851 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:59:15.064851 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:59:15.064851 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:59:15.064851 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:15.064851 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:15.064851 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:15.064851 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:15.064851 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:15.064851 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:15.064851 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:15.064851 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:15.064851 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:15.064851 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:15.064851 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:15.064851 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:15.065672 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:15.065672 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:15.065672 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:15.065672 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:15.065672 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:59:15.065672 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:59:15.065672 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:59:15.065672 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:59:15.065672 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:59:15.065672 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:59:15.065672 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 30 04:59:15.065672 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 04:59:15.065672 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 177/356] 10.222.0.77 () {56 vars in 1099 bytes} [Tue Aug 30 04:59:14 2022] GET /identity/v3/roles/member => generated 84 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:15.066324 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e7d57e6-6351-4a61-84d5-258682a169ec None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:15.095718 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:15.155865 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1f19ec7-41b6-4dbf-996a-82f7acfae00b None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:15.189335 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e7d57e6-6351-4a61-84d5-258682a169ec None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:15.196336 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3855aa4-d30f-4335-b482-846c88df817a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:15.196950 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3855aa4-d30f-4335-b482-846c88df817a None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:15.197230 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3855aa4-d30f-4335-b482-846c88df817a None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:15.197502 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3855aa4-d30f-4335-b482-846c88df817a None admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:15.198162 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3855aa4-d30f-4335-b482-846c88df817a None admin] RBAC: Adding query filter params (name=reader) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:15.198460 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3855aa4-d30f-4335-b482-846c88df817a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:15.199915 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3855aa4-d30f-4335-b482-846c88df817a None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:15.205772 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:15.209420 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 180/357] 10.222.0.77 () {56 vars in 1101 bytes} [Tue Aug 30 04:59:15 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:15.210092 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3855aa4-d30f-4335-b482-846c88df817a None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:15.231637 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15bb0c73-4dfa-451e-9902-bd5f615c9c40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:15.232245 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15bb0c73-4dfa-451e-9902-bd5f615c9c40 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:15.232518 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15bb0c73-4dfa-451e-9902-bd5f615c9c40 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:15.232789 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15bb0c73-4dfa-451e-9902-bd5f615c9c40 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:15.233509 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15bb0c73-4dfa-451e-9902-bd5f615c9c40 None admin] RBAC: Adding query filter params (group.id=b27079fce8234aa482d7e0497689431c, role.id=6f03d86980c34c36b44a8d6764ff4fda, scope.project.id=9c3b4147cbea42e3a97c54416188a48c) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:15.233811 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15bb0c73-4dfa-451e-9902-bd5f615c9c40 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:15.235317 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15bb0c73-4dfa-451e-9902-bd5f615c9c40 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:15.240764 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 178/358] 10.222.0.77 () {56 vars in 1376 bytes} [Tue Aug 30 04:59:15 2022] GET /identity/v3/role_assignments?group.id=b27079fce8234aa482d7e0497689431c&scope.project.id=9c3b4147cbea42e3a97c54416188a48c&role.id=6f03d86980c34c36b44a8d6764ff4fda => generated 260 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:15.241885 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e336929-e1ba-41da-b86d-1a2b0759e705 None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:15.242168 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e336929-e1ba-41da-b86d-1a2b0759e705 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:15.242469 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e336929-e1ba-41da-b86d-1a2b0759e705 None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:15.243885 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 179/359] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:59:15 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:59:15.244454 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e336929-e1ba-41da-b86d-1a2b0759e705 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:15.345744 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3855aa4-d30f-4335-b482-846c88df817a None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:15.360445 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:15.386054 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e336929-e1ba-41da-b86d-1a2b0759e705 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:15.393100 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0aa6767-d247-4470-84ee-8ddda269a462 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:15.393746 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0aa6767-d247-4470-84ee-8ddda269a462 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:15.394029 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0aa6767-d247-4470-84ee-8ddda269a462 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:15.394399 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0aa6767-d247-4470-84ee-8ddda269a462 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:15.406995 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0aa6767-d247-4470-84ee-8ddda269a462 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:59:15.412742 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-d0aa6767-d247-4470-84ee-8ddda269a462 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 30 04:59:15.412742 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:15.412742 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:15.412742 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:15.412742 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:15.412742 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:15.412742 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:15.412742 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:15.412742 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:15.412742 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:15.412742 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:15.412742 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:15.412742 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:59:15.412742 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:59:15.412742 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:59:15.412742 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:59:15.413638 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:59:15.413638 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:59:15.413638 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:59:15.413638 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:59:15.413638 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:15.413638 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:15.413638 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:15.413638 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:15.413638 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:15.413638 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:15.413638 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:15.413638 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:15.413638 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:15.413638 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:15.413638 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:15.413638 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:15.414482 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:15.414482 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:15.414482 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:15.414482 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:15.414482 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:59:15.414482 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:59:15.414482 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:59:15.414482 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:59:15.414482 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:59:15.414482 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:59:15.414482 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 30 04:59:15.414482 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 04:59:15.414482 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 181/360] 10.222.0.77 () {56 vars in 1114 bytes} [Tue Aug 30 04:59:15 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:15.415239 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0aa6767-d247-4470-84ee-8ddda269a462 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:15.421590 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d656463a-1df4-412c-971e-40434c963b00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:15.422193 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d656463a-1df4-412c-971e-40434c963b00 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:15.422474 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d656463a-1df4-412c-971e-40434c963b00 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:15.422740 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d656463a-1df4-412c-971e-40434c963b00 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:15.423510 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d656463a-1df4-412c-971e-40434c963b00 None admin] RBAC: Adding query filter params (group.id=b27079fce8234aa482d7e0497689431c, role.id=3b9d5509a2f7478c9b4989d921daef2a, scope.project.id=9c3b4147cbea42e3a97c54416188a48c) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:15.423797 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d656463a-1df4-412c-971e-40434c963b00 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:15.425230 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d656463a-1df4-412c-971e-40434c963b00 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:15.431742 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 180/361] 10.222.0.77 () {56 vars in 1376 bytes} [Tue Aug 30 04:59:15 2022] GET /identity/v3/role_assignments?group.id=b27079fce8234aa482d7e0497689431c&scope.project.id=9c3b4147cbea42e3a97c54416188a48c&role.id=3b9d5509a2f7478c9b4989d921daef2a => generated 260 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:15.432310 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d656463a-1df4-412c-971e-40434c963b00 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:15.483330 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:15.545410 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0aa6767-d247-4470-84ee-8ddda269a462 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:15.548928 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d656463a-1df4-412c-971e-40434c963b00 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:15.585560 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd7e5479-6e13-4198-a11d-cc358c35aba0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:15.586222 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd7e5479-6e13-4198-a11d-cc358c35aba0 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:15.586512 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd7e5479-6e13-4198-a11d-cc358c35aba0 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:15.586783 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd7e5479-6e13-4198-a11d-cc358c35aba0 None admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:15.587502 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd7e5479-6e13-4198-a11d-cc358c35aba0 None admin] RBAC: Adding query filter params (name=member) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:15.587804 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd7e5479-6e13-4198-a11d-cc358c35aba0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:15.588080 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60a175fa-b45f-44b1-ab2b-4bc160418795 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:15.588863 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60a175fa-b45f-44b1-ab2b-4bc160418795 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:15.589204 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd7e5479-6e13-4198-a11d-cc358c35aba0 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:15.589420 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60a175fa-b45f-44b1-ab2b-4bc160418795 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:15.589836 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60a175fa-b45f-44b1-ab2b-4bc160418795 None admin] PATH_INFO: `/v3/users/800bfa69da174787ad3469d55da101b4` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:15.591198 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60a175fa-b45f-44b1-ab2b-4bc160418795 None admin] RBAC: Authorizing `identity:get_user(user_id=800bfa69da174787ad3469d55da101b4)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:15.592805 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60a175fa-b45f-44b1-ab2b-4bc160418795 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:15.594638 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 181/362] 10.222.0.77 () {56 vars in 1177 bytes} [Tue Aug 30 04:59:15 2022] GET /identity/v3/users/800bfa69da174787ad3469d55da101b4 => generated 293 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:15.595986 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-250bbff7-fb1c-4e61-ae29-a3c99e46b689 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:15.596460 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-250bbff7-fb1c-4e61-ae29-a3c99e46b689 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:15.596896 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-250bbff7-fb1c-4e61-ae29-a3c99e46b689 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:15.598911 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 182/363] 10.222.0.77 () {56 vars in 1101 bytes} [Tue Aug 30 04:59:15 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:15.599753 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd7e5479-6e13-4198-a11d-cc358c35aba0 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:15.623953 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-250bbff7-fb1c-4e61-ae29-a3c99e46b689 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:59:15.625748 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-250bbff7-fb1c-4e61-ae29-a3c99e46b689 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:59:15.644211 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:15.710740 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 182/364] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:59:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:59:15.712967 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-250bbff7-fb1c-4e61-ae29-a3c99e46b689 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:15.732880 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd7e5479-6e13-4198-a11d-cc358c35aba0 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:15.760302 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:15.807403 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-396390da-8fe6-421c-ae64-fd800761da7e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:15.808467 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-396390da-8fe6-421c-ae64-fd800761da7e None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:15.809043 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-396390da-8fe6-421c-ae64-fd800761da7e None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:15.809513 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-396390da-8fe6-421c-ae64-fd800761da7e None admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:15.810377 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-396390da-8fe6-421c-ae64-fd800761da7e None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:15.815058 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-396390da-8fe6-421c-ae64-fd800761da7e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:15.816698 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-396390da-8fe6-421c-ae64-fd800761da7e None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:15.824561 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-250bbff7-fb1c-4e61-ae29-a3c99e46b689 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:15.832400 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 183/365] 10.222.0.77 () {56 vars in 1111 bytes} [Tue Aug 30 04:59:15 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:15.833710 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-396390da-8fe6-421c-ae64-fd800761da7e None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:15.865847 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77ccc96a-4afe-4bdd-b410-efdaa5cb8894 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:15.866450 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77ccc96a-4afe-4bdd-b410-efdaa5cb8894 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:15.866723 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77ccc96a-4afe-4bdd-b410-efdaa5cb8894 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:15.867032 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77ccc96a-4afe-4bdd-b410-efdaa5cb8894 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:15.870866 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77ccc96a-4afe-4bdd-b410-efdaa5cb8894 None admin] RBAC: Adding query filter params (role.id=ddaf2963f700481eb93adea7a36f4e73, user.id=800bfa69da174787ad3469d55da101b4, scope.system=all) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:15.870866 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77ccc96a-4afe-4bdd-b410-efdaa5cb8894 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:15.870866 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77ccc96a-4afe-4bdd-b410-efdaa5cb8894 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:15.885007 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 183/366] 10.222.0.77 () {56 vars in 1308 bytes} [Tue Aug 30 04:59:15 2022] GET /identity/v3/role_assignments?user.id=800bfa69da174787ad3469d55da101b4&scope.system=all&role.id=ddaf2963f700481eb93adea7a36f4e73 => generated 510 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:15.885903 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77ccc96a-4afe-4bdd-b410-efdaa5cb8894 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:16.003266 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:16.004290 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:16.008592 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77ccc96a-4afe-4bdd-b410-efdaa5cb8894 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:16.026386 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-396390da-8fe6-421c-ae64-fd800761da7e None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:16.050077 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6787a7ae-a2e9-4843-bf3d-1816dac1c8b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:16.050984 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6787a7ae-a2e9-4843-bf3d-1816dac1c8b5 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:16.051264 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6787a7ae-a2e9-4843-bf3d-1816dac1c8b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:16.051534 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6787a7ae-a2e9-4843-bf3d-1816dac1c8b5 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:16.075021 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-082ede71-56b3-4325-bc9a-463693640cfd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:16.075647 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-082ede71-56b3-4325-bc9a-463693640cfd None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:16.075926 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-082ede71-56b3-4325-bc9a-463693640cfd None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:16.076539 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-082ede71-56b3-4325-bc9a-463693640cfd None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:16.147640 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6787a7ae-a2e9-4843-bf3d-1816dac1c8b5 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:16.149713 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6787a7ae-a2e9-4843-bf3d-1816dac1c8b5 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:16.173952 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-082ede71-56b3-4325-bc9a-463693640cfd None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:16.176096 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-082ede71-56b3-4325-bc9a-463693640cfd None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:16.284351 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:16.290305 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 184/367] 10.222.0.77 () {58 vars in 1282 bytes} [Tue Aug 30 04:59:15 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 405 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 04:59:16.292044 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6787a7ae-a2e9-4843-bf3d-1816dac1c8b5 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:16.299329 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 184/368] 10.222.0.77 () {58 vars in 1282 bytes} [Tue Aug 30 04:59:15 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 466 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 04:59:16.300179 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-082ede71-56b3-4325-bc9a-463693640cfd None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:16.331208 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:16.405983 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6787a7ae-a2e9-4843-bf3d-1816dac1c8b5 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:16.413137 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-082ede71-56b3-4325-bc9a-463693640cfd None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:16.444279 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51830f81-7074-4210-a999-a59a27bb18dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:16.444928 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51830f81-7074-4210-a999-a59a27bb18dc None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:16.445207 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51830f81-7074-4210-a999-a59a27bb18dc None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:16.445478 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51830f81-7074-4210-a999-a59a27bb18dc None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:16.471416 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-268eb69f-a886-42e3-864d-15c72eb826fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:16.471864 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-268eb69f-a886-42e3-864d-15c72eb826fe None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:16.472140 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-268eb69f-a886-42e3-864d-15c72eb826fe None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:16.472417 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-268eb69f-a886-42e3-864d-15c72eb826fe None admin] PATH_INFO: `/v3/projects/a82de84c774f4bfb8eda1692f1d46a24` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:16.473757 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-268eb69f-a886-42e3-864d-15c72eb826fe None admin] RBAC: Authorizing `identity:get_project(project_id=a82de84c774f4bfb8eda1692f1d46a24)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:16.475251 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-268eb69f-a886-42e3-864d-15c72eb826fe None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:16.475674 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51830f81-7074-4210-a999-a59a27bb18dc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:59:16.477276 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 185/369] 10.222.0.77 () {56 vars in 1186 bytes} [Tue Aug 30 04:59:16 2022] GET /identity/v3/projects/a82de84c774f4bfb8eda1692f1d46a24 => generated 302 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:16.478111 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-268eb69f-a886-42e3-864d-15c72eb826fe None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:16.483359 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-51830f81-7074-4210-a999-a59a27bb18dc None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 04:59:16.483359 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:16.483359 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:16.483359 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:16.483359 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:16.483359 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:16.483359 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:16.483359 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:16.483359 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:16.483359 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:16.483359 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:16.483359 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:16.483359 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:59:16.483359 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:59:16.483359 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:59:16.483359 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:59:16.484238 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:59:16.484238 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:59:16.484238 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:59:16.484238 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:59:16.484238 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:16.484238 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:16.484238 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:16.484238 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:16.484238 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:16.484238 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:16.484238 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:16.484238 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:16.484238 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:16.484238 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:16.484238 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:16.484238 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:16.485012 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:16.485012 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:16.485012 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:16.485012 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:16.485012 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:59:16.485012 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:59:16.485012 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:59:16.485012 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:59:16.485012 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:59:16.485012 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:59:16.485012 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 04:59:16.485012 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 04:59:16.487691 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 185/370] 10.222.0.77 () {56 vars in 1096 bytes} [Tue Aug 30 04:59:16 2022] GET /identity/v3/roles/admin => generated 83 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:16.488687 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51830f81-7074-4210-a999-a59a27bb18dc None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:16.562263 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:16.579185 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:16.623007 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-268eb69f-a886-42e3-864d-15c72eb826fe None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:16.633906 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51830f81-7074-4210-a999-a59a27bb18dc None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:16.664295 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cd8b6e4-4898-427e-818f-9057174ea9eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:16.664967 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd8b6e4-4898-427e-818f-9057174ea9eb None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:16.665231 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd8b6e4-4898-427e-818f-9057174ea9eb None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:16.665491 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd8b6e4-4898-427e-818f-9057174ea9eb None admin] PATH_INFO: `/v3/projects/a82de84c774f4bfb8eda1692f1d46a24` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:16.666650 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cd8b6e4-4898-427e-818f-9057174ea9eb None admin] RBAC: Authorizing `identity:get_project(project_id=a82de84c774f4bfb8eda1692f1d46a24)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:16.669163 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cd8b6e4-4898-427e-818f-9057174ea9eb None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:16.671132 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 186/371] 10.222.0.77 () {56 vars in 1186 bytes} [Tue Aug 30 04:59:16 2022] GET /identity/v3/projects/a82de84c774f4bfb8eda1692f1d46a24 => generated 302 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:16.671896 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cd8b6e4-4898-427e-818f-9057174ea9eb None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:16.682586 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a79b3e89-27c1-4ac8-915b-7c66fca11928 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:16.683909 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a79b3e89-27c1-4ac8-915b-7c66fca11928 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:16.684191 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a79b3e89-27c1-4ac8-915b-7c66fca11928 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:16.684467 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a79b3e89-27c1-4ac8-915b-7c66fca11928 None admin] PATH_INFO: `/v3/groups/b27079fce8234aa482d7e0497689431c` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:16.686131 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a79b3e89-27c1-4ac8-915b-7c66fca11928 None admin] RBAC: Authorizing `identity:get_group(group_id=b27079fce8234aa482d7e0497689431c)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:16.687865 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a79b3e89-27c1-4ac8-915b-7c66fca11928 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:16.689472 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 186/372] 10.222.0.77 () {56 vars in 1180 bytes} [Tue Aug 30 04:59:16 2022] GET /identity/v3/groups/b27079fce8234aa482d7e0497689431c => generated 226 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:16.691119 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a79b3e89-27c1-4ac8-915b-7c66fca11928 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:16.810209 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a79b3e89-27c1-4ac8-915b-7c66fca11928 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:16.812129 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cd8b6e4-4898-427e-818f-9057174ea9eb None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:16.843222 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:16.851022 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:16.851893 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5141d8b-76dd-46db-a80b-18718b70bb8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:16.852772 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5141d8b-76dd-46db-a80b-18718b70bb8a None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:16.853303 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5141d8b-76dd-46db-a80b-18718b70bb8a None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:16.853785 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5141d8b-76dd-46db-a80b-18718b70bb8a None admin] PATH_INFO: `/v3/groups/b27079fce8234aa482d7e0497689431c` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:16.857009 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5141d8b-76dd-46db-a80b-18718b70bb8a None admin] RBAC: Authorizing `identity:get_group(group_id=b27079fce8234aa482d7e0497689431c)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:16.858927 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5141d8b-76dd-46db-a80b-18718b70bb8a None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:16.860032 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b41f41c-2bc0-4b81-8ce1-e1bcd0826aa6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:16.860811 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b41f41c-2bc0-4b81-8ce1-e1bcd0826aa6 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:16.861228 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 187/373] 10.222.0.77 () {56 vars in 1180 bytes} [Tue Aug 30 04:59:16 2022] GET /identity/v3/groups/b27079fce8234aa482d7e0497689431c => generated 226 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:16.861447 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b41f41c-2bc0-4b81-8ce1-e1bcd0826aa6 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:16.861927 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b41f41c-2bc0-4b81-8ce1-e1bcd0826aa6 None admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:16.862364 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5141d8b-76dd-46db-a80b-18718b70bb8a None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:16.862936 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b41f41c-2bc0-4b81-8ce1-e1bcd0826aa6 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:16.863379 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b41f41c-2bc0-4b81-8ce1-e1bcd0826aa6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:16.864871 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b41f41c-2bc0-4b81-8ce1-e1bcd0826aa6 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:16.877446 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 187/374] 10.222.0.77 () {56 vars in 1099 bytes} [Tue Aug 30 04:59:16 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:16.884424 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b41f41c-2bc0-4b81-8ce1-e1bcd0826aa6 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:16.997906 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5141d8b-76dd-46db-a80b-18718b70bb8a None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:17.022173 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b41f41c-2bc0-4b81-8ce1-e1bcd0826aa6 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:17.041751 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abce2ee9-6a70-4f61-8456-6f7377cda6b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:17.042709 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abce2ee9-6a70-4f61-8456-6f7377cda6b5 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:17.043219 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abce2ee9-6a70-4f61-8456-6f7377cda6b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:17.043711 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abce2ee9-6a70-4f61-8456-6f7377cda6b5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:17.044655 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abce2ee9-6a70-4f61-8456-6f7377cda6b5 None admin] RBAC: Adding query filter params (group.id=b27079fce8234aa482d7e0497689431c, role.id=3b9d5509a2f7478c9b4989d921daef2a, scope.project.id=a82de84c774f4bfb8eda1692f1d46a24) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:17.045207 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abce2ee9-6a70-4f61-8456-6f7377cda6b5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:17.046923 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abce2ee9-6a70-4f61-8456-6f7377cda6b5 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:17.052975 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 188/375] 10.222.0.77 () {56 vars in 1376 bytes} [Tue Aug 30 04:59:16 2022] GET /identity/v3/role_assignments?group.id=b27079fce8234aa482d7e0497689431c&scope.project.id=a82de84c774f4bfb8eda1692f1d46a24&role.id=3b9d5509a2f7478c9b4989d921daef2a => generated 260 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:17.054039 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abce2ee9-6a70-4f61-8456-6f7377cda6b5 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:17.079274 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0514c7b-40c2-429f-b64c-6deaa8b9c41d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:17.080298 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0514c7b-40c2-429f-b64c-6deaa8b9c41d None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:17.080782 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0514c7b-40c2-429f-b64c-6deaa8b9c41d None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:17.081277 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0514c7b-40c2-429f-b64c-6deaa8b9c41d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:17.082256 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0514c7b-40c2-429f-b64c-6deaa8b9c41d None admin] RBAC: Adding query filter params (group.id=b27079fce8234aa482d7e0497689431c, role.id=6f03d86980c34c36b44a8d6764ff4fda, scope.project.id=a82de84c774f4bfb8eda1692f1d46a24) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:17.083200 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0514c7b-40c2-429f-b64c-6deaa8b9c41d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:17.086561 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0514c7b-40c2-429f-b64c-6deaa8b9c41d None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:17.099204 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 188/376] 10.222.0.77 () {56 vars in 1376 bytes} [Tue Aug 30 04:59:16 2022] GET /identity/v3/role_assignments?group.id=b27079fce8234aa482d7e0497689431c&scope.project.id=a82de84c774f4bfb8eda1692f1d46a24&role.id=6f03d86980c34c36b44a8d6764ff4fda => generated 260 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:17.145304 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:17.195358 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abce2ee9-6a70-4f61-8456-6f7377cda6b5 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:17.243925 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8342f3b-2ee8-442f-8f09-b5e2878005ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:17.244974 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8342f3b-2ee8-442f-8f09-b5e2878005ad None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:17.245465 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8342f3b-2ee8-442f-8f09-b5e2878005ad None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:17.245943 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8342f3b-2ee8-442f-8f09-b5e2878005ad None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:17.331204 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8342f3b-2ee8-442f-8f09-b5e2878005ad None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:17.333021 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8342f3b-2ee8-442f-8f09-b5e2878005ad None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:17.401784 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:17.459183 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 189/377] 10.222.0.77 () {58 vars in 1282 bytes} [Tue Aug 30 04:59:17 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 405 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 04:59:17.463753 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0514c7b-40c2-429f-b64c-6deaa8b9c41d None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:17.561818 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:17.624136 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0514c7b-40c2-429f-b64c-6deaa8b9c41d None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:17.664042 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89670e9f-7ce9-4cbb-99c0-3eb22a1463df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:17.665215 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89670e9f-7ce9-4cbb-99c0-3eb22a1463df None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:17.665734 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89670e9f-7ce9-4cbb-99c0-3eb22a1463df None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:17.665734 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89670e9f-7ce9-4cbb-99c0-3eb22a1463df None admin] PATH_INFO: `/v3/projects/46478c740f59492585d5babfa714b5b5` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:17.668416 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89670e9f-7ce9-4cbb-99c0-3eb22a1463df None admin] RBAC: Authorizing `identity:get_project(project_id=46478c740f59492585d5babfa714b5b5)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:17.669880 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89670e9f-7ce9-4cbb-99c0-3eb22a1463df None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:17.672827 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 189/378] 10.222.0.77 () {56 vars in 1186 bytes} [Tue Aug 30 04:59:17 2022] GET /identity/v3/projects/46478c740f59492585d5babfa714b5b5 => generated 344 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:17.677533 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8342f3b-2ee8-442f-8f09-b5e2878005ad None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:17.807647 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8342f3b-2ee8-442f-8f09-b5e2878005ad None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:17.810859 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df92c67-dcf6-47e3-b2a7-2d6e7e3cc546 None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:17.810859 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df92c67-dcf6-47e3-b2a7-2d6e7e3cc546 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:17.811289 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df92c67-dcf6-47e3-b2a7-2d6e7e3cc546 None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:17.813834 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 190/379] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:59:17 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:59:17.817839 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ef51fe-34cf-4982-9d92-23615632fa6c None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:17.818377 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ef51fe-34cf-4982-9d92-23615632fa6c None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:17.818881 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ef51fe-34cf-4982-9d92-23615632fa6c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:17.853194 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caac451f-44ee-480d-a522-b6c8c1aac104 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:17.854123 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caac451f-44ee-480d-a522-b6c8c1aac104 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:17.854376 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caac451f-44ee-480d-a522-b6c8c1aac104 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:17.854649 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caac451f-44ee-480d-a522-b6c8c1aac104 None admin] PATH_INFO: `/v3/groups/d1821f59f4f44123a4ebb1c2814e7c2a` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:17.859822 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-07ef51fe-34cf-4982-9d92-23615632fa6c None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:59:17.860073 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caac451f-44ee-480d-a522-b6c8c1aac104 None admin] RBAC: Authorizing `identity:get_group(group_id=d1821f59f4f44123a4ebb1c2814e7c2a)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:17.861465 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caac451f-44ee-480d-a522-b6c8c1aac104 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:17.861731 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-07ef51fe-34cf-4982-9d92-23615632fa6c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:59:17.873805 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 190/380] 10.222.0.77 () {56 vars in 1180 bytes} [Tue Aug 30 04:59:17 2022] GET /identity/v3/groups/d1821f59f4f44123a4ebb1c2814e7c2a => generated 229 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:17.875445 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caac451f-44ee-480d-a522-b6c8c1aac104 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:17.915719 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:17.949416 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 191/381] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:59:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:59:17.951153 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4994e1d6-1d0a-4457-91af-86a6342dbd78 None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:17.951649 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4994e1d6-1d0a-4457-91af-86a6342dbd78 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:17.952120 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4994e1d6-1d0a-4457-91af-86a6342dbd78 None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:17.953860 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 192/382] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:59:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:59:17.958656 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bc9dcbf-1c59-40ab-a754-a1e5a61afb8d None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:17.959199 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bc9dcbf-1c59-40ab-a754-a1e5a61afb8d None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:17.959672 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bc9dcbf-1c59-40ab-a754-a1e5a61afb8d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:17.993472 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-4bc9dcbf-1c59-40ab-a754-a1e5a61afb8d None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:59:17.996027 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-4bc9dcbf-1c59-40ab-a754-a1e5a61afb8d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:59:18.059644 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caac451f-44ee-480d-a522-b6c8c1aac104 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:18.103176 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:18.135799 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69ffe7b5-9e44-465e-828f-65cc76a6424d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:18.136816 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ffe7b5-9e44-465e-828f-65cc76a6424d None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:18.137201 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ffe7b5-9e44-465e-828f-65cc76a6424d None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:18.137508 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ffe7b5-9e44-465e-828f-65cc76a6424d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:18.138541 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69ffe7b5-9e44-465e-828f-65cc76a6424d None admin] RBAC: Adding query filter params (group.id=d1821f59f4f44123a4ebb1c2814e7c2a, role.id=9d9d5d2c270146488249b7ed943280db, scope.project.id=46478c740f59492585d5babfa714b5b5) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:18.138878 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69ffe7b5-9e44-465e-828f-65cc76a6424d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:18.140371 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69ffe7b5-9e44-465e-828f-65cc76a6424d None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:18.141027 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 193/383] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:59:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:59:18.150975 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 191/384] 10.222.0.77 () {56 vars in 1376 bytes} [Tue Aug 30 04:59:17 2022] GET /identity/v3/role_assignments?group.id=d1821f59f4f44123a4ebb1c2814e7c2a&scope.project.id=46478c740f59492585d5babfa714b5b5&role.id=9d9d5d2c270146488249b7ed943280db => generated 260 bytes in 273 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:59:18.351032 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bc9dcbf-1c59-40ab-a754-a1e5a61afb8d None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:18.483504 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bc9dcbf-1c59-40ab-a754-a1e5a61afb8d None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:18.503612 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69ffe7b5-9e44-465e-828f-65cc76a6424d None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:18.540581 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15899bd2-9c17-4574-8c52-637fe0964189 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:18.541238 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15899bd2-9c17-4574-8c52-637fe0964189 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:18.541516 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15899bd2-9c17-4574-8c52-637fe0964189 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:18.541786 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15899bd2-9c17-4574-8c52-637fe0964189 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:18.569770 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15899bd2-9c17-4574-8c52-637fe0964189 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:59:18.575334 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-15899bd2-9c17-4574-8c52-637fe0964189 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 30 04:59:18.575334 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:18.575334 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:18.575334 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:18.575334 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:18.575334 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:18.575334 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:18.575334 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:18.575334 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:18.575334 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:18.575334 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:18.575334 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:18.575334 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:59:18.575334 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:59:18.575334 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:59:18.575334 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:59:18.576214 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:59:18.576214 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:59:18.576214 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:59:18.576214 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:59:18.576214 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:18.576214 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:18.576214 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:18.576214 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:18.576214 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:18.576214 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:18.576214 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:18.576214 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:18.576214 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:18.576214 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:18.576214 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:18.576214 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:18.577003 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:18.577003 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:18.577003 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:18.577003 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:18.577003 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:59:18.577003 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:59:18.577003 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:59:18.577003 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:59:18.577003 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:59:18.577003 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:59:18.577003 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 30 04:59:18.577003 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 04:59:18.581372 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 194/385] 10.222.0.77 () {56 vars in 1114 bytes} [Tue Aug 30 04:59:18 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 226 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:18.582699 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15899bd2-9c17-4574-8c52-637fe0964189 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:18.615262 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:18.628617 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69ffe7b5-9e44-465e-828f-65cc76a6424d None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:18.667781 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:18.720853 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-989705c3-6702-49cb-b083-7a70e10f6def None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:18.722012 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-989705c3-6702-49cb-b083-7a70e10f6def None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:18.722546 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-989705c3-6702-49cb-b083-7a70e10f6def None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:18.723067 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-989705c3-6702-49cb-b083-7a70e10f6def None admin] PATH_INFO: `/v3/roles/member` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:18.738009 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15899bd2-9c17-4574-8c52-637fe0964189 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:18.752454 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-989705c3-6702-49cb-b083-7a70e10f6def None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:59:18.758479 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-989705c3-6702-49cb-b083-7a70e10f6def None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 30 04:59:18.758479 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:18.758479 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:18.758479 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:18.758479 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:18.758479 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:18.758479 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:18.758479 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:18.758479 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:18.758479 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:18.758479 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:18.758479 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:18.758479 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:59:18.758479 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:59:18.758479 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:59:18.758479 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:59:18.759411 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:59:18.759411 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:59:18.759411 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:59:18.759411 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:59:18.759411 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:18.759411 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:18.759411 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:18.759411 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:18.759411 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:18.759411 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:18.759411 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:18.759411 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:18.759411 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:18.759411 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:18.759411 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:18.759411 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:18.760195 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:18.760195 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:18.760195 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:18.760195 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:18.760195 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:59:18.760195 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:59:18.760195 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:59:18.760195 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:59:18.760195 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:59:18.760195 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:59:18.760195 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 30 04:59:18.760195 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 04:59:18.771155 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 192/386] 10.222.0.77 () {56 vars in 1099 bytes} [Tue Aug 30 04:59:18 2022] GET /identity/v3/roles/member => generated 84 bytes in 268 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 04:59:18.772616 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-989705c3-6702-49cb-b083-7a70e10f6def None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:18.784951 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-827a8bb6-70a7-4e04-8fb0-04f71444b006 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:18.785569 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-827a8bb6-70a7-4e04-8fb0-04f71444b006 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:18.785847 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-827a8bb6-70a7-4e04-8fb0-04f71444b006 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:18.786120 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-827a8bb6-70a7-4e04-8fb0-04f71444b006 None admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:18.786797 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-827a8bb6-70a7-4e04-8fb0-04f71444b006 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:18.787123 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-827a8bb6-70a7-4e04-8fb0-04f71444b006 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:18.788523 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-827a8bb6-70a7-4e04-8fb0-04f71444b006 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:18.804673 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 195/387] 10.222.0.77 () {56 vars in 1111 bytes} [Tue Aug 30 04:59:18 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:18.810799 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-827a8bb6-70a7-4e04-8fb0-04f71444b006 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:18.924291 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-827a8bb6-70a7-4e04-8fb0-04f71444b006 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:18.932532 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:18.962365 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e89d56d-502f-43e3-a68e-4faca4168c76 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:18.963161 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e89d56d-502f-43e3-a68e-4faca4168c76 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:18.963506 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e89d56d-502f-43e3-a68e-4faca4168c76 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:18.963851 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e89d56d-502f-43e3-a68e-4faca4168c76 None admin] PATH_INFO: `/v3/groups/b27079fce8234aa482d7e0497689431c` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:18.965581 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e89d56d-502f-43e3-a68e-4faca4168c76 None admin] RBAC: Authorizing `identity:get_group(group_id=b27079fce8234aa482d7e0497689431c)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:18.967369 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e89d56d-502f-43e3-a68e-4faca4168c76 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:18.969647 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 196/388] 10.222.0.77 () {56 vars in 1180 bytes} [Tue Aug 30 04:59:18 2022] GET /identity/v3/groups/b27079fce8234aa482d7e0497689431c => generated 226 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:18.976679 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e89d56d-502f-43e3-a68e-4faca4168c76 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:18.980157 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-989705c3-6702-49cb-b083-7a70e10f6def None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:19.051223 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c776aff6-ab88-4ff2-b89f-bc2e614e2e56 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:19.052217 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c776aff6-ab88-4ff2-b89f-bc2e614e2e56 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:19.052702 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c776aff6-ab88-4ff2-b89f-bc2e614e2e56 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:19.053194 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c776aff6-ab88-4ff2-b89f-bc2e614e2e56 None admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:19.054062 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c776aff6-ab88-4ff2-b89f-bc2e614e2e56 None admin] RBAC: Adding query filter params (name=member) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:19.055453 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c776aff6-ab88-4ff2-b89f-bc2e614e2e56 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:19.057202 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c776aff6-ab88-4ff2-b89f-bc2e614e2e56 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:19.074619 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 193/389] 10.222.0.77 () {56 vars in 1101 bytes} [Tue Aug 30 04:59:18 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 302 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:19.080831 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c776aff6-ab88-4ff2-b89f-bc2e614e2e56 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:19.099921 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:19.168332 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e89d56d-502f-43e3-a68e-4faca4168c76 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:19.243373 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c776aff6-ab88-4ff2-b89f-bc2e614e2e56 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:19.262435 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-342b28d6-d36b-46ff-9839-99152e7c0b4e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:19.264568 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-342b28d6-d36b-46ff-9839-99152e7c0b4e None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:19.264568 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-342b28d6-d36b-46ff-9839-99152e7c0b4e None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:19.264568 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-342b28d6-d36b-46ff-9839-99152e7c0b4e None admin] PATH_INFO: `/v3/projects/9c3b4147cbea42e3a97c54416188a48c` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:19.267506 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-342b28d6-d36b-46ff-9839-99152e7c0b4e None admin] RBAC: Authorizing `identity:get_project(project_id=9c3b4147cbea42e3a97c54416188a48c)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:19.268951 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-342b28d6-d36b-46ff-9839-99152e7c0b4e None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:19.271083 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 197/390] 10.222.0.77 () {56 vars in 1186 bytes} [Tue Aug 30 04:59:18 2022] GET /identity/v3/projects/9c3b4147cbea42e3a97c54416188a48c => generated 298 bytes in 294 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:59:19.277245 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-342b28d6-d36b-46ff-9839-99152e7c0b4e None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:19.283255 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30ba0755-61d8-4a9c-96ad-c6f756b7ba2a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:19.284349 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30ba0755-61d8-4a9c-96ad-c6f756b7ba2a None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:19.284845 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30ba0755-61d8-4a9c-96ad-c6f756b7ba2a None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:19.285340 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30ba0755-61d8-4a9c-96ad-c6f756b7ba2a None admin] PATH_INFO: `/v3/groups/b27079fce8234aa482d7e0497689431c` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:19.286687 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30ba0755-61d8-4a9c-96ad-c6f756b7ba2a None admin] RBAC: Authorizing `identity:get_group(group_id=b27079fce8234aa482d7e0497689431c)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:19.288399 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30ba0755-61d8-4a9c-96ad-c6f756b7ba2a None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:19.290259 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 194/391] 10.222.0.77 () {56 vars in 1180 bytes} [Tue Aug 30 04:59:19 2022] GET /identity/v3/groups/b27079fce8234aa482d7e0497689431c => generated 226 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:59:19.296577 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30ba0755-61d8-4a9c-96ad-c6f756b7ba2a None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:19.353358 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:19.432062 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-342b28d6-d36b-46ff-9839-99152e7c0b4e None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:19.471074 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e52ff32-4ab5-4f85-ab96-b91aa7d16277 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:19.471738 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e52ff32-4ab5-4f85-ab96-b91aa7d16277 None admin] REQUEST_METHOD: `PUT` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:19.472016 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e52ff32-4ab5-4f85-ab96-b91aa7d16277 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:19.472457 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e52ff32-4ab5-4f85-ab96-b91aa7d16277 None admin] PATH_INFO: `/v3/projects/9c3b4147cbea42e3a97c54416188a48c/groups/b27079fce8234aa482d7e0497689431c/roles/3b9d5509a2f7478c9b4989d921daef2a` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:19.475830 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e52ff32-4ab5-4f85-ab96-b91aa7d16277 None admin] RBAC: Authorizing `identity:create_grant(project_id=9c3b4147cbea42e3a97c54416188a48c, group_id=b27079fce8234aa482d7e0497689431c, role_id=3b9d5509a2f7478c9b4989d921daef2a)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:19.477846 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e52ff32-4ab5-4f85-ab96-b91aa7d16277 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:19.496233 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 198/392] 10.222.0.77 () {58 vars in 1442 bytes} [Tue Aug 30 04:59:19 2022] PUT /identity/v3/projects/9c3b4147cbea42e3a97c54416188a48c/groups/b27079fce8234aa482d7e0497689431c/roles/3b9d5509a2f7478c9b4989d921daef2a => generated 0 bytes in 219 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:59:19.519624 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:19.556686 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30ba0755-61d8-4a9c-96ad-c6f756b7ba2a None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:19.576376 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1009cdde-d122-44b4-8045-a45f14306f11 None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:19.576722 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1009cdde-d122-44b4-8045-a45f14306f11 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:19.576988 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1009cdde-d122-44b4-8045-a45f14306f11 None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:19.578750 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 199/393] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:59:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:59:19.588370 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25019183-8aec-4780-a8c3-347cb57aeb59 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:19.588691 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25019183-8aec-4780-a8c3-347cb57aeb59 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:19.588971 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25019183-8aec-4780-a8c3-347cb57aeb59 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:19.619378 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-759af902-f3dc-4c75-be8d-5fcbe0a1dd72 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:19.620028 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-759af902-f3dc-4c75-be8d-5fcbe0a1dd72 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:19.620305 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-759af902-f3dc-4c75-be8d-5fcbe0a1dd72 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:19.620576 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-759af902-f3dc-4c75-be8d-5fcbe0a1dd72 None admin] PATH_INFO: `/v3/projects/9c3b4147cbea42e3a97c54416188a48c` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:19.621912 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-759af902-f3dc-4c75-be8d-5fcbe0a1dd72 None admin] RBAC: Authorizing `identity:get_project(project_id=9c3b4147cbea42e3a97c54416188a48c)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:19.624144 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-759af902-f3dc-4c75-be8d-5fcbe0a1dd72 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:19.627968 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 195/394] 10.222.0.77 () {56 vars in 1186 bytes} [Tue Aug 30 04:59:19 2022] GET /identity/v3/projects/9c3b4147cbea42e3a97c54416188a48c => generated 298 bytes in 332 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:59:19.630016 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-25019183-8aec-4780-a8c3-347cb57aeb59 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:59:19.631299 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-25019183-8aec-4780-a8c3-347cb57aeb59 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:59:19.634321 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-759af902-f3dc-4c75-be8d-5fcbe0a1dd72 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:19.746641 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 200/395] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:59:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:59:19.747878 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b3c94ac-8692-44f0-87d0-a447fdff8dac None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:19.748158 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b3c94ac-8692-44f0-87d0-a447fdff8dac None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:19.748424 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b3c94ac-8692-44f0-87d0-a447fdff8dac None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:19.750034 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 201/396] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:59:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:59:19.755350 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e4293c4-1411-49c9-83d2-b60f3c09ea60 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:19.755636 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e4293c4-1411-49c9-83d2-b60f3c09ea60 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:19.755907 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e4293c4-1411-49c9-83d2-b60f3c09ea60 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:19.770759 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-759af902-f3dc-4c75-be8d-5fcbe0a1dd72 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:19.774805 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-7e4293c4-1411-49c9-83d2-b60f3c09ea60 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:59:19.776032 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-7e4293c4-1411-49c9-83d2-b60f3c09ea60 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:59:19.805196 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:19.817382 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e3daf40-90dd-444c-8876-b676b1194d3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:19.818371 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e3daf40-90dd-444c-8876-b676b1194d3c None admin] REQUEST_METHOD: `PUT` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:19.818659 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e3daf40-90dd-444c-8876-b676b1194d3c None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:19.819917 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e3daf40-90dd-444c-8876-b676b1194d3c None admin] PATH_INFO: `/v3/projects/9c3b4147cbea42e3a97c54416188a48c/groups/b27079fce8234aa482d7e0497689431c/roles/6f03d86980c34c36b44a8d6764ff4fda` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:19.821923 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e3daf40-90dd-444c-8876-b676b1194d3c None admin] RBAC: Authorizing `identity:create_grant(project_id=9c3b4147cbea42e3a97c54416188a48c, group_id=b27079fce8234aa482d7e0497689431c, role_id=6f03d86980c34c36b44a8d6764ff4fda)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:19.823486 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e3daf40-90dd-444c-8876-b676b1194d3c None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:19.842950 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 196/397] 10.222.0.77 () {58 vars in 1442 bytes} [Tue Aug 30 04:59:19 2022] PUT /identity/v3/projects/9c3b4147cbea42e3a97c54416188a48c/groups/b27079fce8234aa482d7e0497689431c/roles/6f03d86980c34c36b44a8d6764ff4fda => generated 0 bytes in 209 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 04:59:19.864932 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 202/398] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:59:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 30 04:59:20.150433 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e3daf40-90dd-444c-8876-b676b1194d3c None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:20.154464 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:20.162595 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e4293c4-1411-49c9-83d2-b60f3c09ea60 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:20.313704 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e4293c4-1411-49c9-83d2-b60f3c09ea60 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:20.353760 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a631ba4b-0dcd-415a-a784-829a4e2221d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:20.354759 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a631ba4b-0dcd-415a-a784-829a4e2221d4 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:20.355279 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a631ba4b-0dcd-415a-a784-829a4e2221d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:20.356035 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a631ba4b-0dcd-415a-a784-829a4e2221d4 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:20.367674 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a631ba4b-0dcd-415a-a784-829a4e2221d4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:59:20.373722 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-a631ba4b-0dcd-415a-a784-829a4e2221d4 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 30 04:59:20.373722 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:20.373722 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:20.373722 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:20.373722 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:20.373722 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:20.373722 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:20.373722 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:20.373722 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:20.373722 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:20.373722 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:20.373722 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:20.373722 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:59:20.373722 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:59:20.373722 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:59:20.373722 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:59:20.374644 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:59:20.374644 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:59:20.374644 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:59:20.374644 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:59:20.374644 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:20.374644 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:20.374644 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:20.374644 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:20.374644 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:20.374644 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:20.374644 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:20.374644 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:20.374644 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:20.374644 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:20.374644 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:20.374644 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:20.375509 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:20.375509 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:20.375509 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:20.375509 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:20.375509 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:59:20.375509 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:59:20.375509 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:59:20.375509 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:59:20.375509 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:59:20.375509 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:59:20.375509 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 30 04:59:20.375509 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 04:59:20.378173 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 203/399] 10.222.0.77 () {56 vars in 1114 bytes} [Tue Aug 30 04:59:20 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:20.383960 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a631ba4b-0dcd-415a-a784-829a4e2221d4 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:20.409116 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e3daf40-90dd-444c-8876-b676b1194d3c None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:20.413057 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:20.508602 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caa2bdd2-dfcd-4cf1-9a91-fd28dd69e113 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:20.509240 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caa2bdd2-dfcd-4cf1-9a91-fd28dd69e113 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:20.509518 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caa2bdd2-dfcd-4cf1-9a91-fd28dd69e113 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:20.509783 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caa2bdd2-dfcd-4cf1-9a91-fd28dd69e113 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:20.513459 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a631ba4b-0dcd-415a-a784-829a4e2221d4 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:20.541932 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caa2bdd2-dfcd-4cf1-9a91-fd28dd69e113 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:59:20.547318 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-caa2bdd2-dfcd-4cf1-9a91-fd28dd69e113 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 30 04:59:20.547318 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:20.547318 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:20.547318 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:20.547318 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:20.547318 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:20.547318 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:20.547318 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:20.547318 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:20.547318 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:20.547318 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:20.547318 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:20.547318 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:59:20.547318 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:59:20.547318 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:59:20.547318 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:59:20.548201 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:59:20.548201 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:59:20.548201 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:59:20.548201 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:59:20.548201 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:20.548201 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:20.548201 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:20.548201 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:20.548201 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:20.548201 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:20.548201 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:20.548201 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:20.548201 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:20.548201 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:20.548201 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:20.548201 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:20.548979 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:20.548979 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:20.548979 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:20.548979 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:20.548979 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:59:20.548979 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:59:20.548979 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:59:20.548979 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:59:20.548979 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:59:20.548979 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:59:20.548979 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 30 04:59:20.548979 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 04:59:20.553436 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbb7d273-eaa4-4a77-8753-45e592811c69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:20.554288 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb7d273-eaa4-4a77-8753-45e592811c69 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:20.554791 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb7d273-eaa4-4a77-8753-45e592811c69 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:20.555951 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb7d273-eaa4-4a77-8753-45e592811c69 None admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:20.556078 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 197/400] 10.222.0.77 () {56 vars in 1099 bytes} [Tue Aug 30 04:59:20 2022] GET /identity/v3/roles/member => generated 84 bytes in 406 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:20.556926 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbb7d273-eaa4-4a77-8753-45e592811c69 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:20.557431 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbb7d273-eaa4-4a77-8753-45e592811c69 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:20.559015 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbb7d273-eaa4-4a77-8753-45e592811c69 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:20.565034 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caa2bdd2-dfcd-4cf1-9a91-fd28dd69e113 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:20.568716 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 204/401] 10.222.0.77 () {56 vars in 1111 bytes} [Tue Aug 30 04:59:20 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:59:20.576523 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbb7d273-eaa4-4a77-8753-45e592811c69 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:20.669988 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:20.682375 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:20.686696 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbb7d273-eaa4-4a77-8753-45e592811c69 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:20.721936 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bfd60d3-92b3-4f61-83ea-c7e09757e0fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:20.722561 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bfd60d3-92b3-4f61-83ea-c7e09757e0fc None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:20.722859 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bfd60d3-92b3-4f61-83ea-c7e09757e0fc None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:20.723139 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bfd60d3-92b3-4f61-83ea-c7e09757e0fc None admin] PATH_INFO: `/v3/groups/b27079fce8234aa482d7e0497689431c` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:20.724258 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bfd60d3-92b3-4f61-83ea-c7e09757e0fc None admin] RBAC: Authorizing `identity:get_group(group_id=b27079fce8234aa482d7e0497689431c)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:20.725685 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bfd60d3-92b3-4f61-83ea-c7e09757e0fc None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:20.727599 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 205/402] 10.222.0.77 () {56 vars in 1180 bytes} [Tue Aug 30 04:59:20 2022] GET /identity/v3/groups/b27079fce8234aa482d7e0497689431c => generated 226 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:59:20.728970 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecbb8e19-110d-4ce9-ad2c-1cead8b7b552 None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:20.729251 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecbb8e19-110d-4ce9-ad2c-1cead8b7b552 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:20.729521 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecbb8e19-110d-4ce9-ad2c-1cead8b7b552 None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:20.729628 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caa2bdd2-dfcd-4cf1-9a91-fd28dd69e113 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:20.735435 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 206/403] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:59:20 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:59:20.736220 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecbb8e19-110d-4ce9-ad2c-1cead8b7b552 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:20.767230 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea98f2d5-4df3-407f-8b77-71cd44807b62 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:20.768214 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea98f2d5-4df3-407f-8b77-71cd44807b62 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:20.768693 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea98f2d5-4df3-407f-8b77-71cd44807b62 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:20.769201 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea98f2d5-4df3-407f-8b77-71cd44807b62 None admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:20.770053 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea98f2d5-4df3-407f-8b77-71cd44807b62 None admin] RBAC: Adding query filter params (name=member) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:20.770644 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea98f2d5-4df3-407f-8b77-71cd44807b62 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:20.772379 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea98f2d5-4df3-407f-8b77-71cd44807b62 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:20.781737 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 198/404] 10.222.0.77 () {56 vars in 1101 bytes} [Tue Aug 30 04:59:20 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:20.783334 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2accab6-8384-457c-94bf-0137412bc38d None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:20.783851 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2accab6-8384-457c-94bf-0137412bc38d None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:20.784322 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2accab6-8384-457c-94bf-0137412bc38d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:20.803312 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-c2accab6-8384-457c-94bf-0137412bc38d None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:59:20.805382 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-c2accab6-8384-457c-94bf-0137412bc38d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:59:20.841663 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecbb8e19-110d-4ce9-ad2c-1cead8b7b552 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:20.875491 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bb81928-6590-4615-a113-a2c7eb747cf0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:20.876196 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb81928-6590-4615-a113-a2c7eb747cf0 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:20.876477 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb81928-6590-4615-a113-a2c7eb747cf0 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:20.876747 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb81928-6590-4615-a113-a2c7eb747cf0 None admin] PATH_INFO: `/v3/projects/a82de84c774f4bfb8eda1692f1d46a24` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:20.877900 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bb81928-6590-4615-a113-a2c7eb747cf0 None admin] RBAC: Authorizing `identity:get_project(project_id=a82de84c774f4bfb8eda1692f1d46a24)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:20.879499 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bb81928-6590-4615-a113-a2c7eb747cf0 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:20.885598 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 199/405] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:59:20 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:59:20.886621 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 207/406] 10.222.0.77 () {56 vars in 1186 bytes} [Tue Aug 30 04:59:20 2022] GET /identity/v3/projects/a82de84c774f4bfb8eda1692f1d46a24 => generated 302 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:20.887229 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2accab6-8384-457c-94bf-0137412bc38d None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:20.888260 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bb81928-6590-4615-a113-a2c7eb747cf0 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:20.915230 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:20.950930 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:20.995623 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bb81928-6590-4615-a113-a2c7eb747cf0 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:20.998990 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2accab6-8384-457c-94bf-0137412bc38d None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:21.029638 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c23f7ff-154d-4d4c-9447-695e73eae89c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:21.030280 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c23f7ff-154d-4d4c-9447-695e73eae89c None admin] REQUEST_METHOD: `PUT` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:21.030560 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c23f7ff-154d-4d4c-9447-695e73eae89c None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:21.030846 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c23f7ff-154d-4d4c-9447-695e73eae89c None admin] PATH_INFO: `/v3/projects/a82de84c774f4bfb8eda1692f1d46a24/groups/b27079fce8234aa482d7e0497689431c/roles/3b9d5509a2f7478c9b4989d921daef2a` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:21.043534 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c23f7ff-154d-4d4c-9447-695e73eae89c None admin] RBAC: Authorizing `identity:create_grant(project_id=a82de84c774f4bfb8eda1692f1d46a24, group_id=b27079fce8234aa482d7e0497689431c, role_id=3b9d5509a2f7478c9b4989d921daef2a)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:21.044449 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a873914-6d70-40e0-9038-74f2d1727a6e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:21.044972 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c23f7ff-154d-4d4c-9447-695e73eae89c None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:21.045530 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a873914-6d70-40e0-9038-74f2d1727a6e None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:21.046014 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a873914-6d70-40e0-9038-74f2d1727a6e None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:21.046488 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a873914-6d70-40e0-9038-74f2d1727a6e None admin] PATH_INFO: `/v3/groups/b27079fce8234aa482d7e0497689431c` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:21.047891 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a873914-6d70-40e0-9038-74f2d1727a6e None admin] RBAC: Authorizing `identity:get_group(group_id=b27079fce8234aa482d7e0497689431c)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:21.049761 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a873914-6d70-40e0-9038-74f2d1727a6e None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:21.051974 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 200/407] 10.222.0.77 () {56 vars in 1180 bytes} [Tue Aug 30 04:59:20 2022] GET /identity/v3/groups/b27079fce8234aa482d7e0497689431c => generated 226 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:21.059709 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a873914-6d70-40e0-9038-74f2d1727a6e None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:21.072538 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 208/408] 10.222.0.77 () {58 vars in 1442 bytes} [Tue Aug 30 04:59:20 2022] PUT /identity/v3/projects/a82de84c774f4bfb8eda1692f1d46a24/groups/b27079fce8234aa482d7e0497689431c/roles/3b9d5509a2f7478c9b4989d921daef2a => generated 0 bytes in 185 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:59:21.175792 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a873914-6d70-40e0-9038-74f2d1727a6e None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:21.204007 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:21.215437 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc343e17-4530-448a-a155-ec983168795a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:21.216538 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc343e17-4530-448a-a155-ec983168795a None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:21.217249 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc343e17-4530-448a-a155-ec983168795a None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:21.217524 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc343e17-4530-448a-a155-ec983168795a None admin] PATH_INFO: `/v3/projects/a82de84c774f4bfb8eda1692f1d46a24` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:21.218735 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc343e17-4530-448a-a155-ec983168795a None admin] RBAC: Authorizing `identity:get_project(project_id=a82de84c774f4bfb8eda1692f1d46a24)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:21.220526 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc343e17-4530-448a-a155-ec983168795a None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:21.222681 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 201/409] 10.222.0.77 () {56 vars in 1186 bytes} [Tue Aug 30 04:59:21 2022] GET /identity/v3/projects/a82de84c774f4bfb8eda1692f1d46a24 => generated 302 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:21.229081 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c23f7ff-154d-4d4c-9447-695e73eae89c None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:21.304558 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc343e17-4530-448a-a155-ec983168795a None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:21.347220 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:21.381844 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c23f7ff-154d-4d4c-9447-695e73eae89c None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:21.429831 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc343e17-4530-448a-a155-ec983168795a None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:21.430148 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef86f86b-bc7f-417a-bd3b-bd2117ce76f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:21.431056 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef86f86b-bc7f-417a-bd3b-bd2117ce76f1 None admin] REQUEST_METHOD: `PUT` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:21.431777 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef86f86b-bc7f-417a-bd3b-bd2117ce76f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:21.432220 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef86f86b-bc7f-417a-bd3b-bd2117ce76f1 None admin] PATH_INFO: `/v3/projects/a82de84c774f4bfb8eda1692f1d46a24/groups/b27079fce8234aa482d7e0497689431c/roles/6f03d86980c34c36b44a8d6764ff4fda` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:21.435391 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef86f86b-bc7f-417a-bd3b-bd2117ce76f1 None admin] RBAC: Authorizing `identity:create_grant(project_id=a82de84c774f4bfb8eda1692f1d46a24, group_id=b27079fce8234aa482d7e0497689431c, role_id=6f03d86980c34c36b44a8d6764ff4fda)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:21.437003 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef86f86b-bc7f-417a-bd3b-bd2117ce76f1 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:21.454441 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 209/410] 10.222.0.77 () {58 vars in 1442 bytes} [Tue Aug 30 04:59:21 2022] PUT /identity/v3/projects/a82de84c774f4bfb8eda1692f1d46a24/groups/b27079fce8234aa482d7e0497689431c/roles/6f03d86980c34c36b44a8d6764ff4fda => generated 0 bytes in 226 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:59:21.499679 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceb235f7-6d1c-4b08-98bd-989354fd6633 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:21.500664 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb235f7-6d1c-4b08-98bd-989354fd6633 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:21.501151 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb235f7-6d1c-4b08-98bd-989354fd6633 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:21.501570 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb235f7-6d1c-4b08-98bd-989354fd6633 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:21.515870 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceb235f7-6d1c-4b08-98bd-989354fd6633 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:59:21.521512 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-ceb235f7-6d1c-4b08-98bd-989354fd6633 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 04:59:21.521512 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:21.521512 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:21.521512 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:21.521512 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:21.521512 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:21.521512 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:21.521512 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:21.521512 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:21.521512 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:21.521512 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:21.521512 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:21.521512 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:59:21.521512 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:59:21.521512 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:59:21.521512 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:59:21.522763 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:59:21.522763 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:59:21.522763 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:59:21.522763 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:59:21.522763 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:21.522763 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:21.522763 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:21.522763 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:21.522763 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:21.522763 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:21.522763 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:21.522763 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:21.522763 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:21.522763 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:21.522763 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:21.522763 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:21.524347 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:21.524347 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:21.524347 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:21.524347 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:21.524347 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:59:21.524347 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:59:21.524347 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:59:21.524347 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:59:21.524347 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:59:21.524347 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:59:21.524347 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 04:59:21.524347 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 04:59:21.526898 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 202/411] 10.222.0.77 () {56 vars in 1096 bytes} [Tue Aug 30 04:59:21 2022] GET /identity/v3/roles/admin => generated 83 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 04:59:21.532528 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef86f86b-bc7f-417a-bd3b-bd2117ce76f1 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:21.662700 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef86f86b-bc7f-417a-bd3b-bd2117ce76f1 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:21.702849 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d1aba98-7340-4c42-b4e8-eb8b7a244507 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:21.703751 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d1aba98-7340-4c42-b4e8-eb8b7a244507 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:21.704178 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d1aba98-7340-4c42-b4e8-eb8b7a244507 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:21.704676 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d1aba98-7340-4c42-b4e8-eb8b7a244507 None admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:21.705511 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d1aba98-7340-4c42-b4e8-eb8b7a244507 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:21.705960 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d1aba98-7340-4c42-b4e8-eb8b7a244507 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:21.707566 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d1aba98-7340-4c42-b4e8-eb8b7a244507 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:21.717783 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 210/412] 10.222.0.77 () {56 vars in 1099 bytes} [Tue Aug 30 04:59:21 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:21.723086 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceb235f7-6d1c-4b08-98bd-989354fd6633 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:21.782076 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:21.822547 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-183d8f98-ba22-4a57-b812-c917c5604745 None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:21.822847 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-183d8f98-ba22-4a57-b812-c917c5604745 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:21.823153 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-183d8f98-ba22-4a57-b812-c917c5604745 None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:21.824752 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 211/413] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:59:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 04:59:21.829758 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3f39e37-dbfe-4009-88a7-0ca49b0b8612 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:21.830039 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3f39e37-dbfe-4009-88a7-0ca49b0b8612 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:21.830315 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3f39e37-dbfe-4009-88a7-0ca49b0b8612 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:21.837268 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceb235f7-6d1c-4b08-98bd-989354fd6633 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:21.837724 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:21.854539 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-b3f39e37-dbfe-4009-88a7-0ca49b0b8612 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:59:21.856571 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-b3f39e37-dbfe-4009-88a7-0ca49b0b8612 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:59:21.881981 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5de90a5-e4a4-4584-b3e6-f494f0619983 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:21.882611 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5de90a5-e4a4-4584-b3e6-f494f0619983 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:21.884342 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5de90a5-e4a4-4584-b3e6-f494f0619983 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:21.884623 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5de90a5-e4a4-4584-b3e6-f494f0619983 None admin] PATH_INFO: `/v3/groups/d1821f59f4f44123a4ebb1c2814e7c2a` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:21.886349 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5de90a5-e4a4-4584-b3e6-f494f0619983 None admin] RBAC: Authorizing `identity:get_group(group_id=d1821f59f4f44123a4ebb1c2814e7c2a)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:21.888671 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5de90a5-e4a4-4584-b3e6-f494f0619983 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:21.899290 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 203/414] 10.222.0.77 () {56 vars in 1180 bytes} [Tue Aug 30 04:59:21 2022] GET /identity/v3/groups/d1821f59f4f44123a4ebb1c2814e7c2a => generated 229 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:21.900617 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5de90a5-e4a4-4584-b3e6-f494f0619983 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:22.041937 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5de90a5-e4a4-4584-b3e6-f494f0619983 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:22.064145 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 212/415] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:59:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 235 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 30 04:59:22.075353 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:22.079480 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c94968c-4a25-4f57-9e3e-21685bb7b14e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:22.080078 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c94968c-4a25-4f57-9e3e-21685bb7b14e None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:22.080363 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c94968c-4a25-4f57-9e3e-21685bb7b14e None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:22.080730 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c94968c-4a25-4f57-9e3e-21685bb7b14e None admin] PATH_INFO: `/v3/projects/46478c740f59492585d5babfa714b5b5` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:22.082014 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c94968c-4a25-4f57-9e3e-21685bb7b14e None admin] RBAC: Authorizing `identity:get_project(project_id=46478c740f59492585d5babfa714b5b5)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:22.083657 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c94968c-4a25-4f57-9e3e-21685bb7b14e None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:22.085691 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 204/416] 10.222.0.77 () {56 vars in 1186 bytes} [Tue Aug 30 04:59:21 2022] GET /identity/v3/projects/46478c740f59492585d5babfa714b5b5 => generated 344 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:22.096215 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3f39e37-dbfe-4009-88a7-0ca49b0b8612 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:22.255969 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3f39e37-dbfe-4009-88a7-0ca49b0b8612 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:22.285893 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:22.291378 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a49cff35-c0aa-4191-b633-1637942fdae6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:22.292532 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a49cff35-c0aa-4191-b633-1637942fdae6 None admin] REQUEST_METHOD: `PUT` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:22.293182 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a49cff35-c0aa-4191-b633-1637942fdae6 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:22.293520 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a49cff35-c0aa-4191-b633-1637942fdae6 None admin] PATH_INFO: `/v3/projects/46478c740f59492585d5babfa714b5b5/groups/d1821f59f4f44123a4ebb1c2814e7c2a/roles/9d9d5d2c270146488249b7ed943280db` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:22.295603 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a49cff35-c0aa-4191-b633-1637942fdae6 None admin] RBAC: Authorizing `identity:create_grant(project_id=46478c740f59492585d5babfa714b5b5, group_id=d1821f59f4f44123a4ebb1c2814e7c2a, role_id=9d9d5d2c270146488249b7ed943280db)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:22.297108 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a49cff35-c0aa-4191-b633-1637942fdae6 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:22.317791 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 213/417] 10.222.0.77 () {58 vars in 1442 bytes} [Tue Aug 30 04:59:22 2022] PUT /identity/v3/projects/46478c740f59492585d5babfa714b5b5/groups/d1821f59f4f44123a4ebb1c2814e7c2a/roles/9d9d5d2c270146488249b7ed943280db => generated 0 bytes in 222 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:59:22.331773 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-639618ff-b33f-4e5b-82be-3f33afcc70c7 None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:22.332326 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-639618ff-b33f-4e5b-82be-3f33afcc70c7 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:22.332326 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-639618ff-b33f-4e5b-82be-3f33afcc70c7 None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:22.333903 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 205/418] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:59:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:59:22.339474 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b10371-9564-4e34-98c2-f57a0686124b None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:22.339894 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b10371-9564-4e34-98c2-f57a0686124b None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:22.340233 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b10371-9564-4e34-98c2-f57a0686124b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:22.363874 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-07b10371-9564-4e34-98c2-f57a0686124b None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:59:22.365275 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-07b10371-9564-4e34-98c2-f57a0686124b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:59:22.395810 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-639618ff-b33f-4e5b-82be-3f33afcc70c7 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:22.459285 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 214/419] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:59:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:59:22.554251 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-639618ff-b33f-4e5b-82be-3f33afcc70c7 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:22.589468 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3cf65c7-dcec-49ff-ba69-8e753eb2fa84 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:22.590136 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3cf65c7-dcec-49ff-ba69-8e753eb2fa84 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:22.590419 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3cf65c7-dcec-49ff-ba69-8e753eb2fa84 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:22.590690 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3cf65c7-dcec-49ff-ba69-8e753eb2fa84 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:22.604409 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3cf65c7-dcec-49ff-ba69-8e753eb2fa84 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:59:22.609284 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-b3cf65c7-dcec-49ff-ba69-8e753eb2fa84 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 30 04:59:22.609284 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:22.609284 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:22.609284 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:22.609284 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:22.609284 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:22.609284 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:22.609284 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:22.609284 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:22.609284 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:22.609284 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:22.609284 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:22.609284 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:59:22.609284 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:59:22.609284 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:59:22.609284 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:59:22.610147 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:59:22.610147 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:59:22.610147 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:59:22.610147 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:59:22.610147 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:22.610147 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:22.610147 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:22.610147 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:22.610147 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:22.610147 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:22.610147 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:22.610147 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:22.610147 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:22.610147 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:22.610147 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:22.610147 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:22.611114 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:22.611114 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:22.611114 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:22.611114 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:22.611114 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:59:22.611114 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:59:22.611114 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:59:22.611114 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:59:22.611114 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:59:22.611114 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:59:22.611114 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 30 04:59:22.611114 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 04:59:22.611114 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 206/420] 10.222.0.77 () {56 vars in 1114 bytes} [Tue Aug 30 04:59:22 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:22.616081 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b10371-9564-4e34-98c2-f57a0686124b None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:22.755793 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b10371-9564-4e34-98c2-f57a0686124b None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:22.767837 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3cf65c7-dcec-49ff-ba69-8e753eb2fa84 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:22.792404 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:22.816519 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3330294c-2112-4275-a8db-503d5dc5d2ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:22.817055 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3330294c-2112-4275-a8db-503d5dc5d2ab None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:22.817333 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3330294c-2112-4275-a8db-503d5dc5d2ab None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:22.817655 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3330294c-2112-4275-a8db-503d5dc5d2ab None admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:22.818417 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3330294c-2112-4275-a8db-503d5dc5d2ab None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:22.818796 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3330294c-2112-4275-a8db-503d5dc5d2ab None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:22.820257 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3330294c-2112-4275-a8db-503d5dc5d2ab None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:22.830134 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:22.834489 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 215/421] 10.222.0.77 () {56 vars in 1111 bytes} [Tue Aug 30 04:59:22 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:59:22.843000 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3330294c-2112-4275-a8db-503d5dc5d2ab None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:22.911416 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3cf65c7-dcec-49ff-ba69-8e753eb2fa84 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:22.984350 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90c04ae0-84a0-4d69-8a9a-8a1c12f1b5c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:22.984975 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90c04ae0-84a0-4d69-8a9a-8a1c12f1b5c8 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:22.985250 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90c04ae0-84a0-4d69-8a9a-8a1c12f1b5c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:22.985512 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90c04ae0-84a0-4d69-8a9a-8a1c12f1b5c8 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:23.007910 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90c04ae0-84a0-4d69-8a9a-8a1c12f1b5c8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:59:23.012948 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-90c04ae0-84a0-4d69-8a9a-8a1c12f1b5c8 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 30 04:59:23.012948 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:23.012948 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:23.012948 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:23.012948 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:23.012948 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:23.012948 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:23.012948 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:23.012948 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:23.012948 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:23.012948 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:23.012948 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:23.012948 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:59:23.012948 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:59:23.012948 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:59:23.012948 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:59:23.014013 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:59:23.014013 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:59:23.014013 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:59:23.014013 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:59:23.014013 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:23.014013 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:23.014013 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:23.014013 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:23.014013 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:23.014013 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:23.014013 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:23.014013 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:23.014013 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:23.014013 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:23.014013 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:23.014013 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:23.014873 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:23.014873 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:23.014873 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:23.014873 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:23.014873 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:59:23.014873 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:59:23.014873 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:59:23.014873 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:59:23.014873 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:59:23.014873 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:59:23.014873 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 30 04:59:23.014873 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 04:59:23.014873 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 207/422] 10.222.0.77 () {56 vars in 1099 bytes} [Tue Aug 30 04:59:22 2022] GET /identity/v3/roles/member => generated 84 bytes in 247 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 04:59:23.024475 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90c04ae0-84a0-4d69-8a9a-8a1c12f1b5c8 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:23.047046 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3330294c-2112-4275-a8db-503d5dc5d2ab None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:23.091761 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7db64a5-3ccb-4b70-9948-5128ce3b2893 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:23.092600 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7db64a5-3ccb-4b70-9948-5128ce3b2893 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:23.093024 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7db64a5-3ccb-4b70-9948-5128ce3b2893 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:23.093454 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7db64a5-3ccb-4b70-9948-5128ce3b2893 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:23.197362 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90c04ae0-84a0-4d69-8a9a-8a1c12f1b5c8 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:23.203141 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:23.214512 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:23.232877 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7db64a5-3ccb-4b70-9948-5128ce3b2893 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:23.234337 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7db64a5-3ccb-4b70-9948-5128ce3b2893 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:23.249714 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e956d6f-09a7-46be-b15d-d7754562b1f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:23.250531 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e956d6f-09a7-46be-b15d-d7754562b1f8 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:23.251255 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e956d6f-09a7-46be-b15d-d7754562b1f8 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:23.253295 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e956d6f-09a7-46be-b15d-d7754562b1f8 None admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:23.254111 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e956d6f-09a7-46be-b15d-d7754562b1f8 None admin] RBAC: Adding query filter params (name=member) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:23.254579 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e956d6f-09a7-46be-b15d-d7754562b1f8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:23.256327 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e956d6f-09a7-46be-b15d-d7754562b1f8 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:23.276110 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 208/423] 10.222.0.77 () {56 vars in 1101 bytes} [Tue Aug 30 04:59:23 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 253 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:23.281540 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e956d6f-09a7-46be-b15d-d7754562b1f8 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:23.401653 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 216/424] 10.222.0.77 () {58 vars in 1282 bytes} [Tue Aug 30 04:59:22 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 559 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Aug 30 04:59:23.411925 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7db64a5-3ccb-4b70-9948-5128ce3b2893 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:23.433445 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e956d6f-09a7-46be-b15d-d7754562b1f8 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:23.472911 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42e74553-9c80-42f8-b322-144d83375f99 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:23.472911 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e74553-9c80-42f8-b322-144d83375f99 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:23.472911 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e74553-9c80-42f8-b322-144d83375f99 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:23.472911 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e74553-9c80-42f8-b322-144d83375f99 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:23.518895 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:23.535190 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:23.552185 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7db64a5-3ccb-4b70-9948-5128ce3b2893 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:23.556677 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42e74553-9c80-42f8-b322-144d83375f99 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:23.558394 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42e74553-9c80-42f8-b322-144d83375f99 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:23.588884 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ded16d8e-742b-4f9c-93e0-9bff04da9842 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:23.589822 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ded16d8e-742b-4f9c-93e0-9bff04da9842 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:23.589993 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ded16d8e-742b-4f9c-93e0-9bff04da9842 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:23.590337 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ded16d8e-742b-4f9c-93e0-9bff04da9842 None admin] PATH_INFO: `/v3/projects/9c3b4147cbea42e3a97c54416188a48c` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:23.591640 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ded16d8e-742b-4f9c-93e0-9bff04da9842 None admin] RBAC: Authorizing `identity:get_project(project_id=9c3b4147cbea42e3a97c54416188a48c)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:23.593161 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ded16d8e-742b-4f9c-93e0-9bff04da9842 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:23.594892 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 217/425] 10.222.0.77 () {56 vars in 1186 bytes} [Tue Aug 30 04:59:23 2022] GET /identity/v3/projects/9c3b4147cbea42e3a97c54416188a48c => generated 298 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:23.596127 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce64f6a-6eaf-409b-8ef4-f632b247c09c None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:23.596473 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce64f6a-6eaf-409b-8ef4-f632b247c09c None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:23.596818 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce64f6a-6eaf-409b-8ef4-f632b247c09c None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:23.598795 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 218/426] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:59:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:59:23.603814 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ce64f6a-6eaf-409b-8ef4-f632b247c09c None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:23.672196 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 209/427] 10.222.0.77 () {58 vars in 1282 bytes} [Tue Aug 30 04:59:23 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 391 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Aug 30 04:59:23.673651 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c11399ec-3f7d-4b14-b3fc-ebe38ef3fbfd None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:23.674095 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c11399ec-3f7d-4b14-b3fc-ebe38ef3fbfd None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:23.674606 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c11399ec-3f7d-4b14-b3fc-ebe38ef3fbfd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:23.693917 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-c11399ec-3f7d-4b14-b3fc-ebe38ef3fbfd None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:59:23.696051 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-c11399ec-3f7d-4b14-b3fc-ebe38ef3fbfd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:59:23.715822 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ce64f6a-6eaf-409b-8ef4-f632b247c09c None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:23.753003 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-449ba187-8190-44e6-989d-be944d943a38 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:23.753632 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449ba187-8190-44e6-989d-be944d943a38 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:23.753958 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449ba187-8190-44e6-989d-be944d943a38 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:23.754233 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449ba187-8190-44e6-989d-be944d943a38 None admin] PATH_INFO: `/v3/groups/b27079fce8234aa482d7e0497689431c` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:23.755369 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-449ba187-8190-44e6-989d-be944d943a38 None admin] RBAC: Authorizing `identity:get_group(group_id=b27079fce8234aa482d7e0497689431c)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:23.756813 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-449ba187-8190-44e6-989d-be944d943a38 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:23.759398 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 219/428] 10.222.0.77 () {56 vars in 1180 bytes} [Tue Aug 30 04:59:23 2022] GET /identity/v3/groups/b27079fce8234aa482d7e0497689431c => generated 226 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:23.760178 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-449ba187-8190-44e6-989d-be944d943a38 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:23.768398 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:23.774234 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 210/429] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:59:23 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:59:23.776013 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c11399ec-3f7d-4b14-b3fc-ebe38ef3fbfd None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:23.791032 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:23.881411 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-449ba187-8190-44e6-989d-be944d943a38 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:23.890301 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c11399ec-3f7d-4b14-b3fc-ebe38ef3fbfd None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:23.926840 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61d92e6a-5419-40c6-ada2-576eaaec66e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:23.927408 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d92e6a-5419-40c6-ada2-576eaaec66e6 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:23.927685 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d92e6a-5419-40c6-ada2-576eaaec66e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:23.927957 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d92e6a-5419-40c6-ada2-576eaaec66e6 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:23.928676 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61d92e6a-5419-40c6-ada2-576eaaec66e6 None admin] RBAC: Adding query filter params (group.id=b27079fce8234aa482d7e0497689431c, role.id=3b9d5509a2f7478c9b4989d921daef2a, scope.project.id=9c3b4147cbea42e3a97c54416188a48c) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:23.928981 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61d92e6a-5419-40c6-ada2-576eaaec66e6 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:23.931076 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61d92e6a-5419-40c6-ada2-576eaaec66e6 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:23.938836 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 211/430] 10.222.0.77 () {56 vars in 1376 bytes} [Tue Aug 30 04:59:23 2022] GET /identity/v3/role_assignments?group.id=b27079fce8234aa482d7e0497689431c&scope.project.id=9c3b4147cbea42e3a97c54416188a48c&role.id=3b9d5509a2f7478c9b4989d921daef2a => generated 611 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:23.945360 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10812b08-97d7-4f95-b58b-1412083e8e69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:23.945360 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10812b08-97d7-4f95-b58b-1412083e8e69 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:23.945360 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10812b08-97d7-4f95-b58b-1412083e8e69 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:23.945360 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10812b08-97d7-4f95-b58b-1412083e8e69 None admin] PATH_INFO: `/v3/projects/9c3b4147cbea42e3a97c54416188a48c` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:23.946212 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10812b08-97d7-4f95-b58b-1412083e8e69 None admin] RBAC: Authorizing `identity:get_project(project_id=9c3b4147cbea42e3a97c54416188a48c)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:23.947830 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10812b08-97d7-4f95-b58b-1412083e8e69 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:23.949455 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 220/431] 10.222.0.77 () {56 vars in 1186 bytes} [Tue Aug 30 04:59:23 2022] GET /identity/v3/projects/9c3b4147cbea42e3a97c54416188a48c => generated 298 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:23.953158 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92a834a3-da5b-4af1-a1e2-791367719cf5 None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:23.953446 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92a834a3-da5b-4af1-a1e2-791367719cf5 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:23.953721 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92a834a3-da5b-4af1-a1e2-791367719cf5 None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:23.956933 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 212/432] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:59:23 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:59:23.958955 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10812b08-97d7-4f95-b58b-1412083e8e69 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:23.964479 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3922e214-03a9-45cc-8d3d-fce46d9c8b42 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:23.964767 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3922e214-03a9-45cc-8d3d-fce46d9c8b42 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:23.965036 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3922e214-03a9-45cc-8d3d-fce46d9c8b42 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:23.998485 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-3922e214-03a9-45cc-8d3d-fce46d9c8b42 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:59:23.999776 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-3922e214-03a9-45cc-8d3d-fce46d9c8b42 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:59:24.103587 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:24.120768 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10812b08-97d7-4f95-b58b-1412083e8e69 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:24.131569 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:24.141544 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 213/433] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:59:23 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 30 04:59:24.150111 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3922e214-03a9-45cc-8d3d-fce46d9c8b42 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:24.172925 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2c5c5bf-d17a-4b30-bd00-b10b0456d657 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:24.173563 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2c5c5bf-d17a-4b30-bd00-b10b0456d657 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:24.173842 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2c5c5bf-d17a-4b30-bd00-b10b0456d657 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:24.174111 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2c5c5bf-d17a-4b30-bd00-b10b0456d657 None admin] PATH_INFO: `/v3/groups/b27079fce8234aa482d7e0497689431c` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:24.175457 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2c5c5bf-d17a-4b30-bd00-b10b0456d657 None admin] RBAC: Authorizing `identity:get_group(group_id=b27079fce8234aa482d7e0497689431c)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:24.177280 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2c5c5bf-d17a-4b30-bd00-b10b0456d657 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:24.179607 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 221/434] 10.222.0.77 () {56 vars in 1180 bytes} [Tue Aug 30 04:59:23 2022] GET /identity/v3/groups/b27079fce8234aa482d7e0497689431c => generated 226 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:24.185856 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2c5c5bf-d17a-4b30-bd00-b10b0456d657 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:24.277916 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3922e214-03a9-45cc-8d3d-fce46d9c8b42 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:24.317615 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b97a7428-f0bd-4e24-8710-f4a8d73ab69e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:24.318238 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b97a7428-f0bd-4e24-8710-f4a8d73ab69e None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:24.318513 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b97a7428-f0bd-4e24-8710-f4a8d73ab69e None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:24.318779 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b97a7428-f0bd-4e24-8710-f4a8d73ab69e None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:24.319387 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2c5c5bf-d17a-4b30-bd00-b10b0456d657 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:24.333939 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b97a7428-f0bd-4e24-8710-f4a8d73ab69e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:59:24.340514 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-b97a7428-f0bd-4e24-8710-f4a8d73ab69e None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 30 04:59:24.340514 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:24.340514 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:24.340514 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:24.340514 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:24.340514 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:24.340514 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:24.340514 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:24.340514 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:24.340514 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:24.340514 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:24.340514 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:24.340514 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:59:24.340514 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:59:24.340514 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:59:24.340514 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:59:24.341415 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:59:24.341415 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:59:24.341415 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:59:24.341415 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:59:24.341415 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:24.341415 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:24.341415 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:24.341415 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:24.341415 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:24.341415 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:24.341415 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:24.341415 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:24.341415 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:24.341415 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:24.341415 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:24.341415 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:24.342192 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:24.342192 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:24.342192 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:24.342192 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:24.342192 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:59:24.342192 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:59:24.342192 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:59:24.342192 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:59:24.342192 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:59:24.342192 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:59:24.342192 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 30 04:59:24.342192 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 04:59:24.342192 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 214/435] 10.222.0.77 () {56 vars in 1114 bytes} [Tue Aug 30 04:59:24 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 04:59:24.350402 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b97a7428-f0bd-4e24-8710-f4a8d73ab69e None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:24.380973 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8f9c0e3-16b4-4276-9aaa-aed57177f46e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:24.381581 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8f9c0e3-16b4-4276-9aaa-aed57177f46e None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:24.381859 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8f9c0e3-16b4-4276-9aaa-aed57177f46e None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:24.382126 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8f9c0e3-16b4-4276-9aaa-aed57177f46e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:24.382935 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8f9c0e3-16b4-4276-9aaa-aed57177f46e None admin] RBAC: Adding query filter params (group.id=b27079fce8234aa482d7e0497689431c, role.id=6f03d86980c34c36b44a8d6764ff4fda, scope.project.id=9c3b4147cbea42e3a97c54416188a48c) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:24.383244 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8f9c0e3-16b4-4276-9aaa-aed57177f46e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:24.385248 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8f9c0e3-16b4-4276-9aaa-aed57177f46e None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:24.398982 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 222/436] 10.222.0.77 () {56 vars in 1376 bytes} [Tue Aug 30 04:59:24 2022] GET /identity/v3/role_assignments?group.id=b27079fce8234aa482d7e0497689431c&scope.project.id=9c3b4147cbea42e3a97c54416188a48c&role.id=6f03d86980c34c36b44a8d6764ff4fda => generated 611 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:24.424087 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:24.467914 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8f9c0e3-16b4-4276-9aaa-aed57177f46e None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:24.478204 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b97a7428-f0bd-4e24-8710-f4a8d73ab69e None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:24.496330 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:24.530909 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e49dc19a-f0a6-4d2d-96aa-52e08c807c9d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:24.531535 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49dc19a-f0a6-4d2d-96aa-52e08c807c9d None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:24.531816 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49dc19a-f0a6-4d2d-96aa-52e08c807c9d None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:24.532104 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49dc19a-f0a6-4d2d-96aa-52e08c807c9d None admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:24.532765 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e49dc19a-f0a6-4d2d-96aa-52e08c807c9d None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:24.533067 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e49dc19a-f0a6-4d2d-96aa-52e08c807c9d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:24.535185 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e49dc19a-f0a6-4d2d-96aa-52e08c807c9d None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:24.548392 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 215/437] 10.222.0.77 () {56 vars in 1111 bytes} [Tue Aug 30 04:59:24 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:24.553662 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e49dc19a-f0a6-4d2d-96aa-52e08c807c9d None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:24.590269 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8f9c0e3-16b4-4276-9aaa-aed57177f46e None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:24.638031 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f5afe7d-76ac-49fb-a889-186432ead9f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:24.638663 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f5afe7d-76ac-49fb-a889-186432ead9f2 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:24.638993 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f5afe7d-76ac-49fb-a889-186432ead9f2 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:24.639265 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f5afe7d-76ac-49fb-a889-186432ead9f2 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:24.654238 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f5afe7d-76ac-49fb-a889-186432ead9f2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:59:24.661077 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-9f5afe7d-76ac-49fb-a889-186432ead9f2 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 30 04:59:24.661077 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:24.661077 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:24.661077 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:24.661077 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:24.661077 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:24.661077 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:24.661077 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:24.661077 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:24.661077 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:24.661077 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:24.661077 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:24.661077 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:59:24.661077 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:59:24.661077 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:59:24.661077 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:59:24.662001 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:59:24.662001 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:59:24.662001 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:59:24.662001 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:59:24.662001 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:24.662001 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:24.662001 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:24.662001 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:24.662001 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:24.662001 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:24.662001 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:24.662001 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:24.662001 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:24.662001 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:24.662001 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:24.662001 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:24.662792 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:24.662792 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:24.662792 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:24.662792 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:24.662792 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:59:24.662792 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:59:24.662792 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:59:24.662792 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:59:24.662792 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:59:24.662792 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:59:24.662792 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 30 04:59:24.662792 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 04:59:24.662792 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 223/438] 10.222.0.77 () {56 vars in 1099 bytes} [Tue Aug 30 04:59:24 2022] GET /identity/v3/roles/member => generated 84 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 04:59:24.667588 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f5afe7d-76ac-49fb-a889-186432ead9f2 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:24.675957 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e49dc19a-f0a6-4d2d-96aa-52e08c807c9d None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:24.698462 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:24.718781 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95e45c17-0114-48ff-95f4-42dba9db97d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:24.719337 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e45c17-0114-48ff-95f4-42dba9db97d4 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:24.719615 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e45c17-0114-48ff-95f4-42dba9db97d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:24.719886 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e45c17-0114-48ff-95f4-42dba9db97d4 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:24.762882 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:24.783320 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f5afe7d-76ac-49fb-a889-186432ead9f2 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:24.804310 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95e45c17-0114-48ff-95f4-42dba9db97d4 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:24.805816 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95e45c17-0114-48ff-95f4-42dba9db97d4 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:24.827810 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f73a535d-5c4d-4341-8e4d-3169c387877f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:24.828427 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f73a535d-5c4d-4341-8e4d-3169c387877f None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:24.828706 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f73a535d-5c4d-4341-8e4d-3169c387877f None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:24.828973 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f73a535d-5c4d-4341-8e4d-3169c387877f None admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:24.829632 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f73a535d-5c4d-4341-8e4d-3169c387877f None admin] RBAC: Adding query filter params (name=member) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:24.829930 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f73a535d-5c4d-4341-8e4d-3169c387877f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:24.831333 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f73a535d-5c4d-4341-8e4d-3169c387877f None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:24.840609 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 224/439] 10.222.0.77 () {56 vars in 1101 bytes} [Tue Aug 30 04:59:24 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:24.841841 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0044e1af-74cb-4483-b408-4c5a63e655cb None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:24.842102 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0044e1af-74cb-4483-b408-4c5a63e655cb None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:24.842373 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0044e1af-74cb-4483-b408-4c5a63e655cb None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:24.843626 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 225/440] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:59:24 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:59:24.846490 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0044e1af-74cb-4483-b408-4c5a63e655cb None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:24.936057 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 216/441] 10.222.0.77 () {58 vars in 1282 bytes} [Tue Aug 30 04:59:24 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 383 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Aug 30 04:59:24.938454 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a9c55c7-59f5-4fd2-aebf-0c71d6429c3d None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:24.938739 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a9c55c7-59f5-4fd2-aebf-0c71d6429c3d None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:24.939024 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a9c55c7-59f5-4fd2-aebf-0c71d6429c3d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:24.959558 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-0a9c55c7-59f5-4fd2-aebf-0c71d6429c3d None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:59:24.960220 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-0a9c55c7-59f5-4fd2-aebf-0c71d6429c3d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:59:24.972772 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:24.986211 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0044e1af-74cb-4483-b408-4c5a63e655cb None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:25.019923 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27816c25-f9c0-421b-96e6-1094de616f4f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:25.020539 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27816c25-f9c0-421b-96e6-1094de616f4f None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:25.020798 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27816c25-f9c0-421b-96e6-1094de616f4f None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:25.021065 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27816c25-f9c0-421b-96e6-1094de616f4f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:25.029635 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 217/442] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:59:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:59:25.030459 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a9c55c7-59f5-4fd2-aebf-0c71d6429c3d None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:25.047961 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:25.110239 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27816c25-f9c0-421b-96e6-1094de616f4f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:25.111722 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27816c25-f9c0-421b-96e6-1094de616f4f None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:25.132918 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a9c55c7-59f5-4fd2-aebf-0c71d6429c3d None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:25.168495 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e23bf5e9-0e8d-4d24-83bd-cdac4b2a70d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:25.169103 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e23bf5e9-0e8d-4d24-83bd-cdac4b2a70d1 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:25.169382 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e23bf5e9-0e8d-4d24-83bd-cdac4b2a70d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:25.169645 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e23bf5e9-0e8d-4d24-83bd-cdac4b2a70d1 None admin] PATH_INFO: `/v3/projects/a82de84c774f4bfb8eda1692f1d46a24` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:25.170742 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e23bf5e9-0e8d-4d24-83bd-cdac4b2a70d1 None admin] RBAC: Authorizing `identity:get_project(project_id=a82de84c774f4bfb8eda1692f1d46a24)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:25.172185 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e23bf5e9-0e8d-4d24-83bd-cdac4b2a70d1 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:25.173809 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 218/443] 10.222.0.77 () {56 vars in 1186 bytes} [Tue Aug 30 04:59:25 2022] GET /identity/v3/projects/a82de84c774f4bfb8eda1692f1d46a24 => generated 302 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:25.179300 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e23bf5e9-0e8d-4d24-83bd-cdac4b2a70d1 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:25.194597 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:25.228689 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 226/444] 10.222.0.77 () {58 vars in 1282 bytes} [Tue Aug 30 04:59:24 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 378 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Aug 30 04:59:25.229890 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27816c25-f9c0-421b-96e6-1094de616f4f None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:25.273738 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:25.285244 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e23bf5e9-0e8d-4d24-83bd-cdac4b2a70d1 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:25.319872 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5e35559-ea91-4d65-a0c0-69e489a149ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:25.321426 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5e35559-ea91-4d65-a0c0-69e489a149ae None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:25.321710 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5e35559-ea91-4d65-a0c0-69e489a149ae None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:25.321979 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5e35559-ea91-4d65-a0c0-69e489a149ae None admin] PATH_INFO: `/v3/groups/b27079fce8234aa482d7e0497689431c` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:25.323129 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5e35559-ea91-4d65-a0c0-69e489a149ae None admin] RBAC: Authorizing `identity:get_group(group_id=b27079fce8234aa482d7e0497689431c)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:25.324559 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5e35559-ea91-4d65-a0c0-69e489a149ae None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:25.326113 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 219/445] 10.222.0.77 () {56 vars in 1180 bytes} [Tue Aug 30 04:59:25 2022] GET /identity/v3/groups/b27079fce8234aa482d7e0497689431c => generated 226 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:59:25.326113 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5e35559-ea91-4d65-a0c0-69e489a149ae None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:25.336796 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27816c25-f9c0-421b-96e6-1094de616f4f None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:25.370511 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69c121f2-2b66-4fa3-b096-3c15279d9086 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:25.371158 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c121f2-2b66-4fa3-b096-3c15279d9086 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:25.371436 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c121f2-2b66-4fa3-b096-3c15279d9086 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:25.371708 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c121f2-2b66-4fa3-b096-3c15279d9086 None admin] PATH_INFO: `/v3/projects/a82de84c774f4bfb8eda1692f1d46a24` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:25.372819 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69c121f2-2b66-4fa3-b096-3c15279d9086 None admin] RBAC: Authorizing `identity:get_project(project_id=a82de84c774f4bfb8eda1692f1d46a24)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:25.374245 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69c121f2-2b66-4fa3-b096-3c15279d9086 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:25.375883 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 227/446] 10.222.0.77 () {56 vars in 1186 bytes} [Tue Aug 30 04:59:25 2022] GET /identity/v3/projects/a82de84c774f4bfb8eda1692f1d46a24 => generated 302 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:25.376638 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69c121f2-2b66-4fa3-b096-3c15279d9086 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:25.411771 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:25.430703 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5e35559-ea91-4d65-a0c0-69e489a149ae None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:25.465455 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dd76dd2-7ec0-4913-adc9-d43dcfaf0d64 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:25.465857 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dd76dd2-7ec0-4913-adc9-d43dcfaf0d64 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:25.466125 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dd76dd2-7ec0-4913-adc9-d43dcfaf0d64 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:25.466400 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dd76dd2-7ec0-4913-adc9-d43dcfaf0d64 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:25.476369 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dd76dd2-7ec0-4913-adc9-d43dcfaf0d64 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:59:25.478262 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69c121f2-2b66-4fa3-b096-3c15279d9086 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:25.482058 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-0dd76dd2-7ec0-4913-adc9-d43dcfaf0d64 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 04:59:25.482058 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:25.482058 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:25.482058 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:25.482058 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:25.482058 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:25.482058 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:25.482058 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:25.482058 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:25.482058 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:25.482058 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:25.482058 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:25.482058 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:59:25.482058 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:59:25.482058 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:59:25.482058 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:59:25.483349 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:59:25.483349 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:59:25.483349 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:59:25.483349 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:59:25.483349 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:25.483349 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:25.483349 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:25.483349 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:25.483349 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:25.483349 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:25.483349 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:25.483349 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:25.483349 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:25.483349 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:25.483349 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:25.483349 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:25.484500 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:25.484500 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:25.484500 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:25.484500 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:25.484500 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:59:25.484500 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:59:25.484500 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:59:25.484500 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:59:25.484500 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:59:25.484500 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:59:25.484500 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 04:59:25.484500 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 04:59:25.484500 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 220/447] 10.222.0.77 () {56 vars in 1096 bytes} [Tue Aug 30 04:59:25 2022] GET /identity/v3/roles/admin => generated 83 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:25.484500 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dd76dd2-7ec0-4913-adc9-d43dcfaf0d64 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:25.489211 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:25.512994 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0de6890f-944d-4126-b211-051191a47f06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:25.513588 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0de6890f-944d-4126-b211-051191a47f06 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:25.513871 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0de6890f-944d-4126-b211-051191a47f06 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:25.514139 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0de6890f-944d-4126-b211-051191a47f06 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:25.514891 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0de6890f-944d-4126-b211-051191a47f06 None admin] RBAC: Adding query filter params (group.id=b27079fce8234aa482d7e0497689431c, role.id=3b9d5509a2f7478c9b4989d921daef2a, scope.project.id=a82de84c774f4bfb8eda1692f1d46a24) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:25.515199 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0de6890f-944d-4126-b211-051191a47f06 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:25.516624 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0de6890f-944d-4126-b211-051191a47f06 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:25.522341 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 228/448] 10.222.0.77 () {56 vars in 1376 bytes} [Tue Aug 30 04:59:25 2022] GET /identity/v3/role_assignments?group.id=b27079fce8234aa482d7e0497689431c&scope.project.id=a82de84c774f4bfb8eda1692f1d46a24&role.id=3b9d5509a2f7478c9b4989d921daef2a => generated 611 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:25.523138 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0de6890f-944d-4126-b211-051191a47f06 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:25.594069 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dd76dd2-7ec0-4913-adc9-d43dcfaf0d64 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:25.631033 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0de6890f-944d-4126-b211-051191a47f06 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:25.632861 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e7ab4b8-920b-4ec3-a4c4-a95f5b045991 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:25.633597 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e7ab4b8-920b-4ec3-a4c4-a95f5b045991 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:25.633942 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e7ab4b8-920b-4ec3-a4c4-a95f5b045991 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:25.634283 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e7ab4b8-920b-4ec3-a4c4-a95f5b045991 None admin] PATH_INFO: `/v3/groups/b27079fce8234aa482d7e0497689431c` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:25.635509 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e7ab4b8-920b-4ec3-a4c4-a95f5b045991 None admin] RBAC: Authorizing `identity:get_group(group_id=b27079fce8234aa482d7e0497689431c)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:25.636978 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e7ab4b8-920b-4ec3-a4c4-a95f5b045991 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:25.638573 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 221/449] 10.222.0.77 () {56 vars in 1180 bytes} [Tue Aug 30 04:59:25 2022] GET /identity/v3/groups/b27079fce8234aa482d7e0497689431c => generated 226 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:25.644664 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e7ab4b8-920b-4ec3-a4c4-a95f5b045991 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:25.657636 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:25.672801 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d189dfb8-6901-499f-941c-509ca8f0719a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:25.673415 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d189dfb8-6901-499f-941c-509ca8f0719a None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:25.673791 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d189dfb8-6901-499f-941c-509ca8f0719a None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:25.674070 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d189dfb8-6901-499f-941c-509ca8f0719a None admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:25.674733 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d189dfb8-6901-499f-941c-509ca8f0719a None admin] RBAC: Adding query filter params (name=admin) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:25.675872 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d189dfb8-6901-499f-941c-509ca8f0719a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:25.676445 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d189dfb8-6901-499f-941c-509ca8f0719a None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:25.687981 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 229/450] 10.222.0.77 () {56 vars in 1099 bytes} [Tue Aug 30 04:59:25 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:25.693678 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d189dfb8-6901-499f-941c-509ca8f0719a None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:25.735641 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:25.760249 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e7ab4b8-920b-4ec3-a4c4-a95f5b045991 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:25.794611 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35c7414e-fe37-4bd8-8ddc-eb89524116c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:25.795233 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35c7414e-fe37-4bd8-8ddc-eb89524116c1 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:25.795515 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35c7414e-fe37-4bd8-8ddc-eb89524116c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:25.795792 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35c7414e-fe37-4bd8-8ddc-eb89524116c1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:25.796506 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35c7414e-fe37-4bd8-8ddc-eb89524116c1 None admin] RBAC: Adding query filter params (group.id=b27079fce8234aa482d7e0497689431c, role.id=6f03d86980c34c36b44a8d6764ff4fda, scope.project.id=a82de84c774f4bfb8eda1692f1d46a24) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:25.796813 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35c7414e-fe37-4bd8-8ddc-eb89524116c1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:25.798252 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35c7414e-fe37-4bd8-8ddc-eb89524116c1 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:25.799157 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d189dfb8-6901-499f-941c-509ca8f0719a None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:25.803852 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 222/451] 10.222.0.77 () {56 vars in 1376 bytes} [Tue Aug 30 04:59:25 2022] GET /identity/v3/role_assignments?group.id=b27079fce8234aa482d7e0497689431c&scope.project.id=a82de84c774f4bfb8eda1692f1d46a24&role.id=6f03d86980c34c36b44a8d6764ff4fda => generated 611 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:59:25.833110 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6f7c579-9751-4116-bd33-5c913011fdcb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:25.833728 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6f7c579-9751-4116-bd33-5c913011fdcb None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:25.834002 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6f7c579-9751-4116-bd33-5c913011fdcb None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:25.834269 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6f7c579-9751-4116-bd33-5c913011fdcb None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:25.906470 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6f7c579-9751-4116-bd33-5c913011fdcb None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:25.907993 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6f7c579-9751-4116-bd33-5c913011fdcb None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:25.942441 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:26.008418 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 230/452] 10.222.0.77 () {58 vars in 1282 bytes} [Tue Aug 30 04:59:25 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 315 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Aug 30 04:59:26.013928 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35c7414e-fe37-4bd8-8ddc-eb89524116c1 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:26.089242 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:26.118605 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35c7414e-fe37-4bd8-8ddc-eb89524116c1 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:26.152450 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54416cb8-acf4-46ed-998d-58a6169828d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:26.153085 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54416cb8-acf4-46ed-998d-58a6169828d3 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:26.153363 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54416cb8-acf4-46ed-998d-58a6169828d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:26.153627 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54416cb8-acf4-46ed-998d-58a6169828d3 None admin] PATH_INFO: `/v3/projects/46478c740f59492585d5babfa714b5b5` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:26.154723 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54416cb8-acf4-46ed-998d-58a6169828d3 None admin] RBAC: Authorizing `identity:get_project(project_id=46478c740f59492585d5babfa714b5b5)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:26.156180 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54416cb8-acf4-46ed-998d-58a6169828d3 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:26.157772 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 223/453] 10.222.0.77 () {56 vars in 1186 bytes} [Tue Aug 30 04:59:26 2022] GET /identity/v3/projects/46478c740f59492585d5babfa714b5b5 => generated 344 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:26.163642 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6f7c579-9751-4116-bd33-5c913011fdcb None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:26.266607 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6f7c579-9751-4116-bd33-5c913011fdcb None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:26.299701 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57af70d5-b579-49c9-9f8b-3c5bf29dd0cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:26.300315 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57af70d5-b579-49c9-9f8b-3c5bf29dd0cc None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:26.300588 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57af70d5-b579-49c9-9f8b-3c5bf29dd0cc None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:26.300855 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57af70d5-b579-49c9-9f8b-3c5bf29dd0cc None admin] PATH_INFO: `/v3/groups/d1821f59f4f44123a4ebb1c2814e7c2a` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:26.301950 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57af70d5-b579-49c9-9f8b-3c5bf29dd0cc None admin] RBAC: Authorizing `identity:get_group(group_id=d1821f59f4f44123a4ebb1c2814e7c2a)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:26.303399 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57af70d5-b579-49c9-9f8b-3c5bf29dd0cc None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:26.304957 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 231/454] 10.222.0.77 () {56 vars in 1180 bytes} [Tue Aug 30 04:59:26 2022] GET /identity/v3/groups/d1821f59f4f44123a4ebb1c2814e7c2a => generated 229 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:26.310877 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54416cb8-acf4-46ed-998d-58a6169828d3 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:26.407969 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54416cb8-acf4-46ed-998d-58a6169828d3 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:26.440560 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a9c5d72-1d7d-48cb-a8e4-bde0604a1dd1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:26.441152 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a9c5d72-1d7d-48cb-a8e4-bde0604a1dd1 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:26.441432 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a9c5d72-1d7d-48cb-a8e4-bde0604a1dd1 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:26.441704 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a9c5d72-1d7d-48cb-a8e4-bde0604a1dd1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:26.442421 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a9c5d72-1d7d-48cb-a8e4-bde0604a1dd1 None admin] RBAC: Adding query filter params (group.id=d1821f59f4f44123a4ebb1c2814e7c2a, role.id=9d9d5d2c270146488249b7ed943280db, scope.project.id=46478c740f59492585d5babfa714b5b5) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:26.442728 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a9c5d72-1d7d-48cb-a8e4-bde0604a1dd1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:26.444183 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a9c5d72-1d7d-48cb-a8e4-bde0604a1dd1 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:26.450288 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 224/455] 10.222.0.77 () {56 vars in 1376 bytes} [Tue Aug 30 04:59:26 2022] GET /identity/v3/role_assignments?group.id=d1821f59f4f44123a4ebb1c2814e7c2a&scope.project.id=46478c740f59492585d5babfa714b5b5&role.id=9d9d5d2c270146488249b7ed943280db => generated 611 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:28.870998 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c44ce595-de74-4c39-86f6-19cde2c0a288 None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:28.871982 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c44ce595-de74-4c39-86f6-19cde2c0a288 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:28.872534 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c44ce595-de74-4c39-86f6-19cde2c0a288 None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:28.874343 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 232/456] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:59:28 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:59:28.879364 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d57324a2-4194-4825-af37-233855bd0b9d None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:28.879779 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d57324a2-4194-4825-af37-233855bd0b9d None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:28.879948 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d57324a2-4194-4825-af37-233855bd0b9d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:28.894029 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:28.900036 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91f22c0d-1768-482c-9141-1b1e8c8e381c None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:28.900561 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91f22c0d-1768-482c-9141-1b1e8c8e381c None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:28.901026 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91f22c0d-1768-482c-9141-1b1e8c8e381c None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:28.902890 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 233/457] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:59:28 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:59:28.903936 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-d57324a2-4194-4825-af37-233855bd0b9d None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:59:28.905988 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-d57324a2-4194-4825-af37-233855bd0b9d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:59:28.907920 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe934189-3a2d-457f-b756-b7b64f56fa88 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:28.908540 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe934189-3a2d-457f-b756-b7b64f56fa88 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:28.909087 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe934189-3a2d-457f-b756-b7b64f56fa88 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:28.921055 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:28.946720 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-fe934189-3a2d-457f-b756-b7b64f56fa88 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:59:28.947977 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-fe934189-3a2d-457f-b756-b7b64f56fa88 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:59:29.008472 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 225/458] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:59:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:59:29.085909 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 234/459] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:59:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 30 04:59:29.203676 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb9e2dea-f8a9-490d-8d44-50a7f87be802 None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:29.204282 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb9e2dea-f8a9-490d-8d44-50a7f87be802 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:29.204778 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb9e2dea-f8a9-490d-8d44-50a7f87be802 None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:29.206596 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 226/460] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:59:29 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:59:29.211481 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29150fd8-79d2-4c0c-98bc-0bc206da19c5 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:29.211771 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29150fd8-79d2-4c0c-98bc-0bc206da19c5 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:29.212049 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29150fd8-79d2-4c0c-98bc-0bc206da19c5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:29.236009 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-29150fd8-79d2-4c0c-98bc-0bc206da19c5 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:59:29.237217 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-29150fd8-79d2-4c0c-98bc-0bc206da19c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:59:29.319162 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 235/461] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:59:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:59:29.373671 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04b9461d-1aff-4698-8f85-ffc4e6c83d86 None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:29.374313 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04b9461d-1aff-4698-8f85-ffc4e6c83d86 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:29.374805 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04b9461d-1aff-4698-8f85-ffc4e6c83d86 None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:29.376381 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 227/462] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:59:29 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:59:29.381685 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ddb4f78-0e5b-4662-9e8e-f081499311d9 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:29.381973 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ddb4f78-0e5b-4662-9e8e-f081499311d9 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:29.382250 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ddb4f78-0e5b-4662-9e8e-f081499311d9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:29.403241 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-2ddb4f78-0e5b-4662-9e8e-f081499311d9 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:59:29.404441 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-2ddb4f78-0e5b-4662-9e8e-f081499311d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:59:29.409452 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04b9461d-1aff-4698-8f85-ffc4e6c83d86 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:29.525834 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:29.533301 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04b9461d-1aff-4698-8f85-ffc4e6c83d86 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:29.537407 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 236/463] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:59:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:59:29.538550 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ddb4f78-0e5b-4662-9e8e-f081499311d9 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:29.576294 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77a4b0d9-c8eb-4fa9-b437-bf8c4ee95fd5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:29.576920 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77a4b0d9-c8eb-4fa9-b437-bf8c4ee95fd5 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:29.577207 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77a4b0d9-c8eb-4fa9-b437-bf8c4ee95fd5 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:29.577482 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77a4b0d9-c8eb-4fa9-b437-bf8c4ee95fd5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:29.582541 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:29.599135 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77a4b0d9-c8eb-4fa9-b437-bf8c4ee95fd5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:29.600601 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77a4b0d9-c8eb-4fa9-b437-bf8c4ee95fd5 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:29.636815 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-77a4b0d9-c8eb-4fa9-b437-bf8c4ee95fd5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:59:29.636815 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:29.636815 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:29.636815 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:29.636815 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:29.636815 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:29.636815 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:29.636815 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:29.636815 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:29.636815 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:29.636815 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:29.636815 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:29.636815 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:59:29.636815 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:59:29.636815 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:59:29.636815 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:59:29.637753 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:29.637753 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:29.637753 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:29.637753 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:29.637753 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:29.637753 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:29.637753 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:29.637753 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:29.637753 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:29.637753 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:29.637753 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:29.637753 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:29.637753 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:29.637753 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:29.637753 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:29.637753 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:29.637753 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:59:29.638590 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:59:29.638590 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:59:29.638590 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 04:59:29.638590 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 228/464] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 04:59:29 2022] GET /identity/v3/domains/Default => generated 87 bytes in 229 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:29.647745 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77a4b0d9-c8eb-4fa9-b437-bf8c4ee95fd5 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:29.673404 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ddb4f78-0e5b-4662-9e8e-f081499311d9 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:29.717463 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bab9960f-46fe-4432-a0e6-5c7b52ca8d3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:29.718219 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bab9960f-46fe-4432-a0e6-5c7b52ca8d3b None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:29.718800 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bab9960f-46fe-4432-a0e6-5c7b52ca8d3b None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:29.719158 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bab9960f-46fe-4432-a0e6-5c7b52ca8d3b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:29.732328 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bab9960f-46fe-4432-a0e6-5c7b52ca8d3b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:29.733776 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bab9960f-46fe-4432-a0e6-5c7b52ca8d3b None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:29.751510 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-bab9960f-46fe-4432-a0e6-5c7b52ca8d3b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:59:29.751510 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:29.751510 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:29.751510 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:29.751510 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:29.751510 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:29.751510 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:29.751510 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:29.751510 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:29.751510 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:29.751510 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:29.751510 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:29.751510 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:59:29.751510 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:59:29.751510 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:59:29.751510 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:59:29.752443 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:29.752443 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:29.752443 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:29.752443 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:29.752443 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:29.752443 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:29.752443 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:29.752443 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:29.752443 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:29.752443 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:29.752443 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:29.752443 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:29.752443 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:29.752443 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:29.752443 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:29.752443 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:29.752443 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:59:29.753262 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:59:29.753262 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:59:29.753262 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 04:59:29.753990 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 237/465] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 04:59:29 2022] GET /identity/v3/domains/Default => generated 87 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:29.754715 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bab9960f-46fe-4432-a0e6-5c7b52ca8d3b None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:29.767259 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77a4b0d9-c8eb-4fa9-b437-bf8c4ee95fd5 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:29.806184 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bee545d-3f6e-410f-bfcc-728512710508 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:29.806786 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bee545d-3f6e-410f-bfcc-728512710508 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:29.807109 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bee545d-3f6e-410f-bfcc-728512710508 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:29.807380 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bee545d-3f6e-410f-bfcc-728512710508 None admin] PATH_INFO: `/v3/domains` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:29.808016 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bee545d-3f6e-410f-bfcc-728512710508 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:29.808319 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bee545d-3f6e-410f-bfcc-728512710508 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:29.809699 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bee545d-3f6e-410f-bfcc-728512710508 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:29.819027 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:29.825976 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 229/466] 10.222.0.77 () {56 vars in 1109 bytes} [Tue Aug 30 04:59:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:29.827035 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bee545d-3f6e-410f-bfcc-728512710508 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:29.880841 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bab9960f-46fe-4432-a0e6-5c7b52ca8d3b None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:29.888584 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:29.920323 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9724593f-3c9b-41b3-b20a-c4787f83890b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:29.921257 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9724593f-3c9b-41b3-b20a-c4787f83890b None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:29.921782 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9724593f-3c9b-41b3-b20a-c4787f83890b None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:29.922036 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9724593f-3c9b-41b3-b20a-c4787f83890b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:29.933722 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9724593f-3c9b-41b3-b20a-c4787f83890b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:29.935196 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9724593f-3c9b-41b3-b20a-c4787f83890b None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:29.946659 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bee545d-3f6e-410f-bfcc-728512710508 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:29.949620 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-9724593f-3c9b-41b3-b20a-c4787f83890b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:59:29.949620 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:29.949620 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:29.949620 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:29.949620 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:29.949620 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:29.949620 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:29.949620 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:29.949620 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:29.949620 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:29.949620 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:29.949620 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:29.949620 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:59:29.949620 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:59:29.949620 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:59:29.949620 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:59:29.950564 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:29.950564 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:29.950564 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:29.950564 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:29.950564 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:29.950564 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:29.950564 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:29.950564 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:29.950564 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:29.950564 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:29.950564 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:29.950564 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:29.950564 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:29.950564 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:29.950564 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:29.950564 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:29.950564 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:59:29.951439 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:59:29.951439 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:59:29.951439 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 04:59:29.951439 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 238/467] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 04:59:29 2022] GET /identity/v3/domains/Default => generated 87 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:29.951439 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9724593f-3c9b-41b3-b20a-c4787f83890b None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:29.990422 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48808647-6484-437b-b439-ddf3b4c12e70 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:29.991350 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48808647-6484-437b-b439-ddf3b4c12e70 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:29.991808 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48808647-6484-437b-b439-ddf3b4c12e70 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:29.992830 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48808647-6484-437b-b439-ddf3b4c12e70 None admin] PATH_INFO: `/v3/domains` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:29.993666 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48808647-6484-437b-b439-ddf3b4c12e70 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:29.993970 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48808647-6484-437b-b439-ddf3b4c12e70 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:29.995377 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48808647-6484-437b-b439-ddf3b4c12e70 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:30.011302 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 230/468] 10.222.0.77 () {56 vars in 1109 bytes} [Tue Aug 30 04:59:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:30.011302 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48808647-6484-437b-b439-ddf3b4c12e70 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:30.061853 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:30.067039 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9724593f-3c9b-41b3-b20a-c4787f83890b None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:30.105054 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a5a78fb-a6f8-4255-a038-dde4abe290ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:30.105827 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a5a78fb-a6f8-4255-a038-dde4abe290ba None admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:30.106108 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a5a78fb-a6f8-4255-a038-dde4abe290ba None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:30.106429 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a5a78fb-a6f8-4255-a038-dde4abe290ba None admin] PATH_INFO: `/v3/users` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:30.107109 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a5a78fb-a6f8-4255-a038-dde4abe290ba None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:30.108770 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a5a78fb-a6f8-4255-a038-dde4abe290ba None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:30.117879 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48808647-6484-437b-b439-ddf3b4c12e70 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:30.128227 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:30.134147 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 239/469] 10.222.0.77 () {60 vars in 1132 bytes} [Tue Aug 30 04:59:29 2022] POST /identity/v3/users => generated 248 bytes in 183 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:59:30.135068 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a5a78fb-a6f8-4255-a038-dde4abe290ba None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:30.167743 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f67b0e9-57b1-4b3f-a642-2cfd7dec704c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:30.168344 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f67b0e9-57b1-4b3f-a642-2cfd7dec704c None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:30.168632 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f67b0e9-57b1-4b3f-a642-2cfd7dec704c None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:30.168961 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f67b0e9-57b1-4b3f-a642-2cfd7dec704c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:30.183044 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f67b0e9-57b1-4b3f-a642-2cfd7dec704c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:30.184100 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f67b0e9-57b1-4b3f-a642-2cfd7dec704c None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:30.201247 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-5f67b0e9-57b1-4b3f-a642-2cfd7dec704c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:59:30.201247 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:30.201247 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:30.201247 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:30.201247 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:30.201247 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:30.201247 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:30.201247 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:30.201247 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:30.201247 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:30.201247 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:30.201247 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:30.201247 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:59:30.201247 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:59:30.201247 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:59:30.201247 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:59:30.202230 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:30.202230 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:30.202230 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:30.202230 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:30.202230 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:30.202230 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:30.202230 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:30.202230 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:30.202230 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:30.202230 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:30.202230 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:30.202230 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:30.202230 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:30.202230 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:30.202230 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:30.202230 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:30.202230 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:59:30.203101 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:59:30.203101 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:59:30.203101 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 04:59:30.203101 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 231/470] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 04:59:30 2022] GET /identity/v3/domains/Default => generated 87 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:30.203768 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f67b0e9-57b1-4b3f-a642-2cfd7dec704c None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:30.262536 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a5a78fb-a6f8-4255-a038-dde4abe290ba None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:30.304846 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55bd5e59-5c7b-4e84-9f0b-2e2afe0604e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:30.305839 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55bd5e59-5c7b-4e84-9f0b-2e2afe0604e2 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:30.306223 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55bd5e59-5c7b-4e84-9f0b-2e2afe0604e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:30.306507 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55bd5e59-5c7b-4e84-9f0b-2e2afe0604e2 None admin] PATH_INFO: `/v3/domains` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:30.307287 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55bd5e59-5c7b-4e84-9f0b-2e2afe0604e2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:30.307599 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55bd5e59-5c7b-4e84-9f0b-2e2afe0604e2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:30.309027 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55bd5e59-5c7b-4e84-9f0b-2e2afe0604e2 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:30.319156 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f67b0e9-57b1-4b3f-a642-2cfd7dec704c None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:30.325090 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 240/471] 10.222.0.77 () {56 vars in 1109 bytes} [Tue Aug 30 04:59:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:30.326191 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55bd5e59-5c7b-4e84-9f0b-2e2afe0604e2 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:30.349653 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:30.365629 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-427a44fd-3a49-435b-b8e6-227bed3a8a78 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:30.366397 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-427a44fd-3a49-435b-b8e6-227bed3a8a78 None admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:30.366736 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-427a44fd-3a49-435b-b8e6-227bed3a8a78 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:30.367427 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-427a44fd-3a49-435b-b8e6-227bed3a8a78 None admin] PATH_INFO: `/v3/users` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:30.368403 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-427a44fd-3a49-435b-b8e6-227bed3a8a78 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:30.370125 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-427a44fd-3a49-435b-b8e6-227bed3a8a78 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:30.397567 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 232/472] 10.222.0.77 () {60 vars in 1132 bytes} [Tue Aug 30 04:59:30 2022] POST /identity/v3/users => generated 246 bytes in 193 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:59:30.397871 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-427a44fd-3a49-435b-b8e6-227bed3a8a78 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:30.431315 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:30.471542 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55bd5e59-5c7b-4e84-9f0b-2e2afe0604e2 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:30.521008 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3086d09-0f28-4704-bdb6-a488945b61ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:30.521657 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3086d09-0f28-4704-bdb6-a488945b61ba None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:30.521941 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3086d09-0f28-4704-bdb6-a488945b61ba None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:30.522246 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3086d09-0f28-4704-bdb6-a488945b61ba None admin] PATH_INFO: `/v3/domains` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:30.522943 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3086d09-0f28-4704-bdb6-a488945b61ba None admin] RBAC: Adding query filter params (name=Default) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:30.523247 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3086d09-0f28-4704-bdb6-a488945b61ba None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:30.525178 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3086d09-0f28-4704-bdb6-a488945b61ba None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:30.539713 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 241/473] 10.222.0.77 () {56 vars in 1109 bytes} [Tue Aug 30 04:59:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:30.540268 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-427a44fd-3a49-435b-b8e6-227bed3a8a78 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:30.541034 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-985da97d-cf16-480c-8c26-056aa8637889 None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:30.541300 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-985da97d-cf16-480c-8c26-056aa8637889 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:30.541574 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-985da97d-cf16-480c-8c26-056aa8637889 None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:30.543173 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 242/474] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:59:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:59:30.549327 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-985da97d-cf16-480c-8c26-056aa8637889 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:30.580471 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f363282-8bd6-4822-aede-dd83062e510a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:30.581131 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f363282-8bd6-4822-aede-dd83062e510a None admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:30.581358 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f363282-8bd6-4822-aede-dd83062e510a None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:30.581728 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f363282-8bd6-4822-aede-dd83062e510a None admin] PATH_INFO: `/v3/users` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:30.582264 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f363282-8bd6-4822-aede-dd83062e510a None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:30.583726 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f363282-8bd6-4822-aede-dd83062e510a None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:30.610691 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 233/475] 10.222.0.77 () {60 vars in 1132 bytes} [Tue Aug 30 04:59:30 2022] POST /identity/v3/users => generated 248 bytes in 213 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:59:30.612273 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dec31ca-1af0-43f2-bfe0-f62aa974c895 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:30.612761 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dec31ca-1af0-43f2-bfe0-f62aa974c895 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:30.613191 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dec31ca-1af0-43f2-bfe0-f62aa974c895 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:30.634380 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-2dec31ca-1af0-43f2-bfe0-f62aa974c895 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:59:30.636040 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-2dec31ca-1af0-43f2-bfe0-f62aa974c895 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:59:30.651204 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:30.688670 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-985da97d-cf16-480c-8c26-056aa8637889 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:30.718844 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 234/476] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:59:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:59:30.736768 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b62f2d0b-9878-4f87-9db0-150544f27b62 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:30.737761 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b62f2d0b-9878-4f87-9db0-150544f27b62 None admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:30.738210 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b62f2d0b-9878-4f87-9db0-150544f27b62 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:30.738664 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b62f2d0b-9878-4f87-9db0-150544f27b62 None admin] PATH_INFO: `/v3/users` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:30.739608 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b62f2d0b-9878-4f87-9db0-150544f27b62 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:30.741257 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b62f2d0b-9878-4f87-9db0-150544f27b62 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:30.767638 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 243/477] 10.222.0.77 () {60 vars in 1132 bytes} [Tue Aug 30 04:59:30 2022] POST /identity/v3/users => generated 249 bytes in 219 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:59:30.994370 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dec31ca-1af0-43f2-bfe0-f62aa974c895 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:30.996599 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:31.104756 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dec31ca-1af0-43f2-bfe0-f62aa974c895 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:31.148509 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2be28a08-00e1-4c76-8c58-46525ea2e79a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:31.149106 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be28a08-00e1-4c76-8c58-46525ea2e79a None admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:31.149380 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be28a08-00e1-4c76-8c58-46525ea2e79a None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:31.149647 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be28a08-00e1-4c76-8c58-46525ea2e79a None admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:31.150307 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2be28a08-00e1-4c76-8c58-46525ea2e79a None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:31.151690 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2be28a08-00e1-4c76-8c58-46525ea2e79a None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:31.165494 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.sql.core [None req-2be28a08-00e1-4c76-8c58-46525ea2e79a None admin] Conflict role: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Aug 30 04:59:31.165494 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Aug 30 04:59:31.165494 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [parameters: {'id': '62e304bdbd5d4dbfa33780fbca171e35', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Aug 30 04:59:31.165494 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81624) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Aug 30 04:59:31.186072 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-2be28a08-00e1-4c76-8c58-46525ea2e79a None admin] Conflict occurred attempting to store role - Duplicate entry found with name anotherrole.: keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Aug 30 04:59:31.186072 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:31.186072 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Aug 30 04:59:31.186072 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application self.dialect.do_execute( Aug 30 04:59:31.186072 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Aug 30 04:59:31.186072 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Aug 30 04:59:31.186072 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Aug 30 04:59:31.186072 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application result = self._query(query) Aug 30 04:59:31.186072 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Aug 30 04:59:31.186072 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application conn.query(q) Aug 30 04:59:31.186072 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Aug 30 04:59:31.186072 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Aug 30 04:59:31.186072 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Aug 30 04:59:31.186072 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application result.read() Aug 30 04:59:31.186072 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Aug 30 04:59:31.187009 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Aug 30 04:59:31.187009 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Aug 30 04:59:31.187009 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application packet.raise_for_error() Aug 30 04:59:31.187009 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Aug 30 04:59:31.187009 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Aug 30 04:59:31.187009 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Aug 30 04:59:31.187009 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Aug 30 04:59:31.187009 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Aug 30 04:59:31.187009 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 04:59:31.187009 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Aug 30 04:59:31.187009 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 04:59:31.187009 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:31.187009 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Aug 30 04:59:31.187009 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return method(*args, **kwargs) Aug 30 04:59:31.187009 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 33, in create_role Aug 30 04:59:31.187009 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return ref.to_dict() Aug 30 04:59:31.187009 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Aug 30 04:59:31.187009 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application next(self.gen) Aug 30 04:59:31.187898 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Aug 30 04:59:31.187898 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application yield resource Aug 30 04:59:31.187898 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Aug 30 04:59:31.187898 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application next(self.gen) Aug 30 04:59:31.187898 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Aug 30 04:59:31.187898 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application self.session.rollback() Aug 30 04:59:31.187898 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Aug 30 04:59:31.187898 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application self.force_reraise() Aug 30 04:59:31.187898 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Aug 30 04:59:31.187898 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise self.value Aug 30 04:59:31.187898 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Aug 30 04:59:31.187898 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Aug 30 04:59:31.187898 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Aug 30 04:59:31.187898 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application session.commit() Aug 30 04:59:31.187898 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Aug 30 04:59:31.187898 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Aug 30 04:59:31.187898 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Aug 30 04:59:31.188743 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application self._prepare_impl() Aug 30 04:59:31.188743 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Aug 30 04:59:31.188743 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application self.session.flush() Aug 30 04:59:31.188743 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Aug 30 04:59:31.188743 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application self._flush(objects) Aug 30 04:59:31.188743 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Aug 30 04:59:31.188743 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Aug 30 04:59:31.188743 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Aug 30 04:59:31.188743 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application compat.raise_( Aug 30 04:59:31.188743 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Aug 30 04:59:31.188743 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception Aug 30 04:59:31.188743 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Aug 30 04:59:31.188743 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application flush_context.execute() Aug 30 04:59:31.188743 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Aug 30 04:59:31.188743 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rec.execute(self) Aug 30 04:59:31.188743 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Aug 30 04:59:31.188743 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Aug 30 04:59:31.189635 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Aug 30 04:59:31.189635 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application _emit_insert_statements( Aug 30 04:59:31.189635 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Aug 30 04:59:31.189635 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application c = connection._execute_20( Aug 30 04:59:31.189635 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Aug 30 04:59:31.189635 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Aug 30 04:59:31.189635 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Aug 30 04:59:31.189635 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Aug 30 04:59:31.189635 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Aug 30 04:59:31.189635 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ret = self._execute_context( Aug 30 04:59:31.189635 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Aug 30 04:59:31.189635 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Aug 30 04:59:31.189635 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Aug 30 04:59:31.189635 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Aug 30 04:59:31.189635 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Aug 30 04:59:31.189635 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception Aug 30 04:59:31.190433 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Aug 30 04:59:31.190433 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application self.dialect.do_execute( Aug 30 04:59:31.190433 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Aug 30 04:59:31.190433 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Aug 30 04:59:31.190433 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Aug 30 04:59:31.190433 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application result = self._query(query) Aug 30 04:59:31.190433 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Aug 30 04:59:31.190433 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application conn.query(q) Aug 30 04:59:31.190433 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Aug 30 04:59:31.190433 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Aug 30 04:59:31.190433 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Aug 30 04:59:31.190433 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application result.read() Aug 30 04:59:31.190433 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Aug 30 04:59:31.190433 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Aug 30 04:59:31.190433 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Aug 30 04:59:31.190433 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application packet.raise_for_error() Aug 30 04:59:31.192886 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Aug 30 04:59:31.192886 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Aug 30 04:59:31.192886 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Aug 30 04:59:31.192886 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Aug 30 04:59:31.192886 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Aug 30 04:59:31.192886 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Aug 30 04:59:31.192886 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [parameters: {'id': '62e304bdbd5d4dbfa33780fbca171e35', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Aug 30 04:59:31.192886 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Aug 30 04:59:31.192886 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 04:59:31.192886 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 04:59:31.192886 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 04:59:31.192886 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:31.192886 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:31.192886 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:31.192886 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:31.192886 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:31.194792 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:31.194792 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:31.194792 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:31.194792 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:31.194792 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:31.194792 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:31.194792 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 115, in post Aug 30 04:59:31.194792 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ref = PROVIDERS.role_api.create_role( Aug 30 04:59:31.194792 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:31.194792 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:31.194792 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1298, in create_role Aug 30 04:59:31.194792 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ret = self.driver.create_role(role_id, role) Aug 30 04:59:31.194792 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Aug 30 04:59:31.194792 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Aug 30 04:59:31.194792 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Aug 30 04:59:31.194792 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 04:59:31.202642 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 235/478] 10.222.0.77 () {60 vars in 1131 bytes} [Tue Aug 30 04:59:30 2022] POST /identity/v3/roles => generated 144 bytes in 194 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:31.203233 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b62f2d0b-9878-4f87-9db0-150544f27b62 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:31.316292 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b62f2d0b-9878-4f87-9db0-150544f27b62 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:31.350261 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:31.355492 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1567d98-efbe-4f72-a7ae-b9a3ac0cfc25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:31.356501 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1567d98-efbe-4f72-a7ae-b9a3ac0cfc25 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:31.356849 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1567d98-efbe-4f72-a7ae-b9a3ac0cfc25 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:31.357119 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1567d98-efbe-4f72-a7ae-b9a3ac0cfc25 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:31.368933 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1567d98-efbe-4f72-a7ae-b9a3ac0cfc25 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:59:31.374121 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-d1567d98-efbe-4f72-a7ae-b9a3ac0cfc25 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 30 04:59:31.374121 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:31.374121 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:31.374121 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:31.374121 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:31.374121 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:31.374121 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:31.374121 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:31.374121 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:31.374121 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:31.374121 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:31.374121 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:31.374121 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:59:31.374121 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:59:31.374121 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:59:31.374121 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:59:31.376211 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:59:31.376211 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:59:31.376211 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:59:31.376211 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:59:31.376211 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:31.376211 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:31.376211 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:31.376211 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:31.376211 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:31.376211 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:31.376211 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:31.376211 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:31.376211 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:31.376211 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:31.376211 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:31.376211 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:31.378124 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:31.378124 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:31.378124 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:31.378124 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:31.378124 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:59:31.378124 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:59:31.378124 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:59:31.378124 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:59:31.378124 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:59:31.378124 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:59:31.378124 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 30 04:59:31.378124 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 04:59:31.378124 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 244/479] 10.222.0.77 () {56 vars in 1114 bytes} [Tue Aug 30 04:59:31 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:31.381129 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2be28a08-00e1-4c76-8c58-46525ea2e79a None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:31.449976 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:31.486539 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2be28a08-00e1-4c76-8c58-46525ea2e79a None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:31.522914 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dd18200-f036-4515-8d8a-9eceb7421a73 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:31.523570 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dd18200-f036-4515-8d8a-9eceb7421a73 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:31.523845 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dd18200-f036-4515-8d8a-9eceb7421a73 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:31.524110 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dd18200-f036-4515-8d8a-9eceb7421a73 None admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:31.524763 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dd18200-f036-4515-8d8a-9eceb7421a73 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:31.525058 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dd18200-f036-4515-8d8a-9eceb7421a73 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:31.526408 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dd18200-f036-4515-8d8a-9eceb7421a73 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:31.539462 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 236/480] 10.222.0.77 () {56 vars in 1111 bytes} [Tue Aug 30 04:59:31 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:32.416252 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a0c9435-6d71-4b98-97c0-d696f8bf106b None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:32.416764 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a0c9435-6d71-4b98-97c0-d696f8bf106b None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:32.416764 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a0c9435-6d71-4b98-97c0-d696f8bf106b None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:32.418383 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 245/481] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:59:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:59:32.423716 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49eda5e8-a6b4-4e2d-98f6-d9cd43d25476 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:32.424118 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49eda5e8-a6b4-4e2d-98f6-d9cd43d25476 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:32.424306 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49eda5e8-a6b4-4e2d-98f6-d9cd43d25476 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:32.443322 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-49eda5e8-a6b4-4e2d-98f6-d9cd43d25476 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:59:32.444508 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-49eda5e8-a6b4-4e2d-98f6-d9cd43d25476 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:59:32.512418 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 237/482] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:59:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:59:32.846637 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a0c9435-6d71-4b98-97c0-d696f8bf106b None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:32.929064 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd3a52f2-8da1-4c3f-9c94-571dd7cdaf91 None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:32.929407 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd3a52f2-8da1-4c3f-9c94-571dd7cdaf91 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:32.929648 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd3a52f2-8da1-4c3f-9c94-571dd7cdaf91 None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:32.931366 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 238/483] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:59:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 04:59:32.936319 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0f604e7-1e00-404a-871b-3064036e4fca None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:32.936646 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0f604e7-1e00-404a-871b-3064036e4fca None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:32.936921 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0f604e7-1e00-404a-871b-3064036e4fca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:32.956184 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-a0f604e7-1e00-404a-871b-3064036e4fca None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:59:32.957368 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-a0f604e7-1e00-404a-871b-3064036e4fca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:59:32.984062 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a0c9435-6d71-4b98-97c0-d696f8bf106b None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:33.044619 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:33.049074 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1a06a28-51be-4137-9937-53ff233e45a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:33.049734 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a06a28-51be-4137-9937-53ff233e45a8 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:33.050009 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a06a28-51be-4137-9937-53ff233e45a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:33.050274 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a06a28-51be-4137-9937-53ff233e45a8 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:33.072769 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1a06a28-51be-4137-9937-53ff233e45a8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:59:33.073603 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 239/484] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:59:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 30 04:59:33.078311 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-c1a06a28-51be-4137-9937-53ff233e45a8 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 30 04:59:33.078311 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:33.078311 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:33.078311 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:33.078311 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:33.078311 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:33.078311 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:33.078311 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:33.078311 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:33.078311 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:33.078311 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:33.078311 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:33.078311 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:59:33.078311 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:59:33.078311 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:59:33.078311 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:59:33.079204 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:59:33.079204 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:59:33.079204 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:59:33.079204 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:59:33.079204 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:33.079204 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:33.079204 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:33.079204 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:33.079204 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:33.079204 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:33.079204 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:33.079204 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:33.079204 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:33.079204 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:33.079204 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:33.079204 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:33.079997 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:33.079997 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:33.079997 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:33.079997 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:33.079997 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:59:33.079997 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:59:33.079997 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:59:33.079997 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:59:33.079997 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:59:33.079997 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:59:33.079997 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 30 04:59:33.079997 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 04:59:33.079997 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 246/485] 10.222.0.77 () {56 vars in 1102 bytes} [Tue Aug 30 04:59:32 2022] GET /identity/v3/roles/service => generated 85 bytes in 234 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:33.087899 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0f604e7-1e00-404a-871b-3064036e4fca None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:33.198902 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb1aa58c-8ea9-47d8-958a-7486ebce71c3 None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:33.199214 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb1aa58c-8ea9-47d8-958a-7486ebce71c3 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:33.199387 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb1aa58c-8ea9-47d8-958a-7486ebce71c3 None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:33.200992 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 247/486] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:59:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:59:33.208260 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4dee213-ecb8-4a5a-b430-d63de2d05999 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:33.208260 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4dee213-ecb8-4a5a-b430-d63de2d05999 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:33.208260 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4dee213-ecb8-4a5a-b430-d63de2d05999 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:33.229867 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-c4dee213-ecb8-4a5a-b430-d63de2d05999 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:59:33.232013 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-c4dee213-ecb8-4a5a-b430-d63de2d05999 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:59:33.265629 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0f604e7-1e00-404a-871b-3064036e4fca None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:33.291375 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:33.307540 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad4ebfb3-1764-47a6-bab5-c24865c7c12c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:33.308454 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad4ebfb3-1764-47a6-bab5-c24865c7c12c None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:33.308946 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad4ebfb3-1764-47a6-bab5-c24865c7c12c None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:33.309408 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad4ebfb3-1764-47a6-bab5-c24865c7c12c None admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:33.310262 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad4ebfb3-1764-47a6-bab5-c24865c7c12c None admin] RBAC: Adding query filter params (name=service) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:33.310785 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad4ebfb3-1764-47a6-bab5-c24865c7c12c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:33.312558 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad4ebfb3-1764-47a6-bab5-c24865c7c12c None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:33.322722 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 248/487] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:59:33 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 30 04:59:33.322722 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa20690a-4d4b-4e72-944d-6b28d3145274 None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:33.322722 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa20690a-4d4b-4e72-944d-6b28d3145274 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:33.322722 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa20690a-4d4b-4e72-944d-6b28d3145274 None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:33.327342 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 249/488] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:59:33 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:59:33.328877 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca8325a7-7b7b-4604-b480-b08735c17de2 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:33.329168 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca8325a7-7b7b-4604-b480-b08735c17de2 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:33.329563 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca8325a7-7b7b-4604-b480-b08735c17de2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:33.331593 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 240/489] 10.222.0.77 () {56 vars in 1103 bytes} [Tue Aug 30 04:59:33 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:33.337059 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad4ebfb3-1764-47a6-bab5-c24865c7c12c None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:33.349741 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-ca8325a7-7b7b-4604-b480-b08735c17de2 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:59:33.350986 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-ca8325a7-7b7b-4604-b480-b08735c17de2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:59:33.406741 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:33.434104 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 250/490] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:59:33 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:59:33.434786 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca8325a7-7b7b-4604-b480-b08735c17de2 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:33.488621 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad4ebfb3-1764-47a6-bab5-c24865c7c12c None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:33.573643 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b948518-117a-40be-919e-08201fefa277 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:33.574270 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b948518-117a-40be-919e-08201fefa277 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:33.574547 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b948518-117a-40be-919e-08201fefa277 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:33.574827 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b948518-117a-40be-919e-08201fefa277 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:33.590943 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b948518-117a-40be-919e-08201fefa277 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:33.592754 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b948518-117a-40be-919e-08201fefa277 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:33.599419 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca8325a7-7b7b-4604-b480-b08735c17de2 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:33.615906 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:33.625407 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-4b948518-117a-40be-919e-08201fefa277 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:59:33.625407 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:33.625407 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:33.625407 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:33.625407 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:33.625407 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:33.625407 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:33.625407 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:33.625407 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:33.625407 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:33.625407 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:33.625407 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:33.625407 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:59:33.625407 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:59:33.625407 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:59:33.625407 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:59:33.626311 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:33.626311 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:33.626311 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:33.626311 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:33.626311 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:33.626311 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:33.626311 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:33.626311 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:33.626311 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:33.626311 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:33.626311 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:33.626311 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:33.626311 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:33.626311 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:33.626311 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:33.626311 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:33.626311 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:59:33.627195 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:59:33.627195 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:59:33.627195 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 04:59:33.630855 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 241/491] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 04:59:33 2022] GET /identity/v3/domains/Default => generated 87 bytes in 291 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 04:59:33.635024 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b948518-117a-40be-919e-08201fefa277 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:33.657251 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af7b2e3d-3cb9-4411-92af-d3ab8e9c217f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:33.658387 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af7b2e3d-3cb9-4411-92af-d3ab8e9c217f None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:33.658669 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af7b2e3d-3cb9-4411-92af-d3ab8e9c217f None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:33.658981 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af7b2e3d-3cb9-4411-92af-d3ab8e9c217f None admin] PATH_INFO: `/v3/roles/service` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:33.673771 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af7b2e3d-3cb9-4411-92af-d3ab8e9c217f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:59:33.680183 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-af7b2e3d-3cb9-4411-92af-d3ab8e9c217f None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 30 04:59:33.680183 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:33.680183 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:33.680183 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:33.680183 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:33.680183 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:33.680183 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:33.680183 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:33.680183 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:33.680183 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:33.680183 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:33.680183 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:33.680183 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:59:33.680183 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:59:33.680183 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:59:33.680183 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:59:33.682179 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:59:33.682179 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:59:33.682179 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:59:33.682179 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:59:33.682179 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:33.682179 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:33.682179 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:33.682179 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:33.682179 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:33.682179 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:33.682179 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:33.682179 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:33.682179 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:33.682179 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:33.682179 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:33.682179 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:33.683057 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:33.683057 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:33.683057 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:33.683057 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:33.683057 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:59:33.683057 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:59:33.683057 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:59:33.683057 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:59:33.683057 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:59:33.683057 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:59:33.683057 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 30 04:59:33.683057 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 04:59:33.683057 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 251/492] 10.222.0.77 () {56 vars in 1102 bytes} [Tue Aug 30 04:59:33 2022] GET /identity/v3/roles/service => generated 85 bytes in 247 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:33.683057 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af7b2e3d-3cb9-4411-92af-d3ab8e9c217f None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:33.751899 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b948518-117a-40be-919e-08201fefa277 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:33.753357 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:33.789473 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-690d96f9-d891-4444-87fa-6925f8da3079 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:33.790080 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-690d96f9-d891-4444-87fa-6925f8da3079 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:33.790364 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-690d96f9-d891-4444-87fa-6925f8da3079 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:33.790635 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-690d96f9-d891-4444-87fa-6925f8da3079 None admin] PATH_INFO: `/v3/domains` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:33.791455 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-690d96f9-d891-4444-87fa-6925f8da3079 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:33.791767 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-690d96f9-d891-4444-87fa-6925f8da3079 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:33.793137 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-690d96f9-d891-4444-87fa-6925f8da3079 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:33.800904 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af7b2e3d-3cb9-4411-92af-d3ab8e9c217f None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:33.806841 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 242/493] 10.222.0.77 () {56 vars in 1109 bytes} [Tue Aug 30 04:59:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:33.807929 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-690d96f9-d891-4444-87fa-6925f8da3079 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:33.837215 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-569d6cb9-eca7-465f-bc4e-261240aab474 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:33.837889 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-569d6cb9-eca7-465f-bc4e-261240aab474 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:33.838095 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-569d6cb9-eca7-465f-bc4e-261240aab474 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:33.838357 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-569d6cb9-eca7-465f-bc4e-261240aab474 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:33.849796 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-569d6cb9-eca7-465f-bc4e-261240aab474 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:59:33.857186 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-569d6cb9-eca7-465f-bc4e-261240aab474 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 30 04:59:33.857186 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:33.857186 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:33.857186 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:33.857186 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:33.857186 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:33.857186 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:33.857186 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:33.857186 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:33.857186 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:33.857186 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:33.857186 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:33.857186 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:59:33.857186 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:59:33.857186 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:59:33.857186 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:59:33.858038 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:59:33.858038 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:59:33.858038 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:59:33.858038 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:59:33.858038 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:33.858038 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:33.858038 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:33.858038 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:33.858038 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:33.858038 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:33.858038 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:33.858038 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:33.858038 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:33.858038 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:33.858038 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:33.858038 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:33.858863 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:33.858863 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:33.858863 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:33.858863 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:33.858863 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:59:33.858863 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:59:33.858863 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:59:33.858863 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:59:33.858863 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:59:33.858863 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:59:33.858863 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 30 04:59:33.858863 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 04:59:33.858863 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 252/494] 10.222.0.77 () {56 vars in 1102 bytes} [Tue Aug 30 04:59:33 2022] GET /identity/v3/roles/service => generated 85 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:33.860017 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-569d6cb9-eca7-465f-bc4e-261240aab474 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:33.896632 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:33.918303 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-690d96f9-d891-4444-87fa-6925f8da3079 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:33.958811 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f636e42f-d119-4706-8fb6-5592e9cdcbaf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:33.959460 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f636e42f-d119-4706-8fb6-5592e9cdcbaf None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:33.959733 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f636e42f-d119-4706-8fb6-5592e9cdcbaf None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:33.960000 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f636e42f-d119-4706-8fb6-5592e9cdcbaf None admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:33.960647 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f636e42f-d119-4706-8fb6-5592e9cdcbaf None admin] RBAC: Adding query filter params (name=service) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:33.960971 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f636e42f-d119-4706-8fb6-5592e9cdcbaf None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:33.962356 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f636e42f-d119-4706-8fb6-5592e9cdcbaf None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:33.974753 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 243/495] 10.222.0.77 () {56 vars in 1103 bytes} [Tue Aug 30 04:59:33 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:33.974753 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f636e42f-d119-4706-8fb6-5592e9cdcbaf None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:33.979688 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-569d6cb9-eca7-465f-bc4e-261240aab474 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:33.997520 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:34.018035 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea176cc6-5ebb-4ef2-ba6f-8019883f6df8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:34.018580 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea176cc6-5ebb-4ef2-ba6f-8019883f6df8 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:34.018878 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea176cc6-5ebb-4ef2-ba6f-8019883f6df8 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:34.019146 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea176cc6-5ebb-4ef2-ba6f-8019883f6df8 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:34.030224 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea176cc6-5ebb-4ef2-ba6f-8019883f6df8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:59:34.035021 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-ea176cc6-5ebb-4ef2-ba6f-8019883f6df8 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 30 04:59:34.035021 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:34.035021 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:34.035021 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:34.035021 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:34.035021 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:34.035021 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:34.035021 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:34.035021 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:34.035021 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:34.035021 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:34.035021 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:34.035021 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:59:34.035021 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:59:34.035021 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:59:34.035021 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:59:34.036063 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:59:34.036063 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:59:34.036063 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:59:34.036063 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:59:34.036063 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:34.036063 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:34.036063 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:34.036063 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:34.036063 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:34.036063 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:34.036063 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:34.036063 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:34.036063 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:34.036063 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:34.036063 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:34.036063 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:34.036848 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:34.036848 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:34.036848 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:34.036848 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:34.036848 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:59:34.036848 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:59:34.036848 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:59:34.036848 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:59:34.036848 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:59:34.036848 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:59:34.036848 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 30 04:59:34.036848 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 04:59:34.036848 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 253/496] 10.222.0.77 () {56 vars in 1102 bytes} [Tue Aug 30 04:59:33 2022] GET /identity/v3/roles/service => generated 85 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:34.037489 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea176cc6-5ebb-4ef2-ba6f-8019883f6df8 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:34.084074 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f636e42f-d119-4706-8fb6-5592e9cdcbaf None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:34.120237 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d29b7db0-e795-4ae3-9999-f3263d35d18b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:34.120855 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d29b7db0-e795-4ae3-9999-f3263d35d18b None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:34.121132 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d29b7db0-e795-4ae3-9999-f3263d35d18b None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:34.121429 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d29b7db0-e795-4ae3-9999-f3263d35d18b None admin] PATH_INFO: `/v3/users/glance` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:34.129526 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d29b7db0-e795-4ae3-9999-f3263d35d18b None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:34.131005 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d29b7db0-e795-4ae3-9999-f3263d35d18b None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:34.135491 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:34.144583 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-d29b7db0-e795-4ae3-9999-f3263d35d18b None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Aug 30 04:59:34.144583 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:34.144583 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:34.144583 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:34.144583 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:34.144583 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:34.144583 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:34.144583 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:34.144583 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:34.144583 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:34.144583 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:34.144583 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:34.144583 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 04:59:34.144583 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 04:59:34.144583 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 04:59:34.144583 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 04:59:34.145501 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:34.145501 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:34.145501 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 04:59:34.145501 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:59:34.145501 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 04:59:34.145501 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:59:34.145501 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:34.145501 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:34.145501 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:34.145501 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:34.145501 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:34.145501 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:34.145501 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:34.145501 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:34.145501 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:34.145501 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:34.145501 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:34.145501 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:34.146467 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:34.146467 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:34.146467 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 04:59:34.146467 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 04:59:34.146467 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 04:59:34.146467 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 04:59:34.146467 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 04:59:34.146467 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 04:59:34.146467 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Aug 30 04:59:34.146467 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 04:59:34.147022 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea176cc6-5ebb-4ef2-ba6f-8019883f6df8 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:34.147090 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 244/497] 10.222.0.77 () {56 vars in 1099 bytes} [Tue Aug 30 04:59:33 2022] GET /identity/v3/users/glance => generated 84 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:34.147090 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d29b7db0-e795-4ae3-9999-f3263d35d18b None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:34.187856 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-739550b2-e821-4a90-bd81-c66478fb173c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:34.188458 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-739550b2-e821-4a90-bd81-c66478fb173c None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:34.188735 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-739550b2-e821-4a90-bd81-c66478fb173c None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:34.189004 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-739550b2-e821-4a90-bd81-c66478fb173c None admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:34.189663 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-739550b2-e821-4a90-bd81-c66478fb173c None admin] RBAC: Adding query filter params (name=service) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:34.189966 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-739550b2-e821-4a90-bd81-c66478fb173c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:34.191367 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-739550b2-e821-4a90-bd81-c66478fb173c None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:34.199280 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 254/498] 10.222.0.77 () {56 vars in 1103 bytes} [Tue Aug 30 04:59:34 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:34.200048 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-739550b2-e821-4a90-bd81-c66478fb173c None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:34.239539 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:34.255773 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d29b7db0-e795-4ae3-9999-f3263d35d18b None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:34.288335 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36c6fed0-b382-442a-be2d-12631ed8599a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:34.289481 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36c6fed0-b382-442a-be2d-12631ed8599a None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:34.289757 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36c6fed0-b382-442a-be2d-12631ed8599a None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:34.290019 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36c6fed0-b382-442a-be2d-12631ed8599a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:34.297614 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-739550b2-e821-4a90-bd81-c66478fb173c None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:34.300856 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36c6fed0-b382-442a-be2d-12631ed8599a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:34.302300 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36c6fed0-b382-442a-be2d-12631ed8599a None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:34.316331 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-36c6fed0-b382-442a-be2d-12631ed8599a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:59:34.316331 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:34.316331 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:34.316331 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:34.316331 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:34.316331 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:34.316331 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:34.316331 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:34.316331 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:34.316331 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:34.316331 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:34.316331 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:34.316331 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:59:34.316331 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:59:34.316331 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:59:34.316331 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:59:34.317193 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:34.317193 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:34.317193 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:34.317193 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:34.317193 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:34.317193 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:34.317193 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:34.317193 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:34.317193 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:34.317193 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:34.317193 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:34.317193 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:34.317193 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:34.317193 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:34.317193 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:34.317193 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:34.317193 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:59:34.318020 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:59:34.318020 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:59:34.318020 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 04:59:34.318020 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 245/499] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 04:59:34 2022] GET /identity/v3/domains/Default => generated 87 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:34.318242 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36c6fed0-b382-442a-be2d-12631ed8599a None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:34.332163 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c3b5aa6-cdb6-4283-8f9c-8027e8ca047a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:34.332764 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c3b5aa6-cdb6-4283-8f9c-8027e8ca047a None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:34.333046 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c3b5aa6-cdb6-4283-8f9c-8027e8ca047a None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:34.333319 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c3b5aa6-cdb6-4283-8f9c-8027e8ca047a None admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:34.333975 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c3b5aa6-cdb6-4283-8f9c-8027e8ca047a None admin] RBAC: Adding query filter params (name=service) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:34.334277 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c3b5aa6-cdb6-4283-8f9c-8027e8ca047a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:34.335682 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c3b5aa6-cdb6-4283-8f9c-8027e8ca047a None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:34.344205 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 255/500] 10.222.0.77 () {56 vars in 1103 bytes} [Tue Aug 30 04:59:34 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:34.348788 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d35f3f30-62b2-4bfc-93a7-379df91090b9 None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:34.348788 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d35f3f30-62b2-4bfc-93a7-379df91090b9 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:34.348788 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d35f3f30-62b2-4bfc-93a7-379df91090b9 None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:34.348788 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 256/501] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:59:34 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:59:34.348788 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d35f3f30-62b2-4bfc-93a7-379df91090b9 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:34.368381 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:34.422770 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36c6fed0-b382-442a-be2d-12631ed8599a None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:34.453772 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d35f3f30-62b2-4bfc-93a7-379df91090b9 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:34.456158 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4dcedeb-eea8-47d9-bdb0-0c0b8e14174b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:34.456757 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4dcedeb-eea8-47d9-bdb0-0c0b8e14174b None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:34.457037 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4dcedeb-eea8-47d9-bdb0-0c0b8e14174b None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:34.457305 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4dcedeb-eea8-47d9-bdb0-0c0b8e14174b None admin] PATH_INFO: `/v3/users` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:34.458030 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4dcedeb-eea8-47d9-bdb0-0c0b8e14174b None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:34.458327 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4dcedeb-eea8-47d9-bdb0-0c0b8e14174b None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:34.459786 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4dcedeb-eea8-47d9-bdb0-0c0b8e14174b None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:34.472250 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:34.490919 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6a485e3-aa4e-4b60-94ec-e44e4762fd61 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:34.491533 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6a485e3-aa4e-4b60-94ec-e44e4762fd61 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:34.491807 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6a485e3-aa4e-4b60-94ec-e44e4762fd61 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:34.492070 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6a485e3-aa4e-4b60-94ec-e44e4762fd61 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:34.497105 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 246/502] 10.222.0.77 () {56 vars in 1137 bytes} [Tue Aug 30 04:59:34 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:34.497915 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4dcedeb-eea8-47d9-bdb0-0c0b8e14174b None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:34.502828 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6a485e3-aa4e-4b60-94ec-e44e4762fd61 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:34.504271 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6a485e3-aa4e-4b60-94ec-e44e4762fd61 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:34.518796 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-a6a485e3-aa4e-4b60-94ec-e44e4762fd61 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:59:34.518796 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:34.518796 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:34.518796 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:34.518796 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:34.518796 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:34.518796 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:34.518796 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:34.518796 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:34.518796 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:34.518796 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:34.518796 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:34.518796 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:59:34.518796 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:59:34.518796 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:59:34.518796 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:59:34.519705 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:34.519705 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:34.519705 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:34.519705 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:34.519705 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:34.519705 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:34.519705 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:34.519705 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:34.519705 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:34.519705 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:34.519705 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:34.519705 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:34.519705 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:34.519705 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:34.519705 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:34.519705 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:34.519705 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:59:34.520527 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:59:34.520527 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:59:34.520527 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 04:59:34.520527 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 257/503] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 04:59:34 2022] GET /identity/v3/domains/Default => generated 87 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:34.521138 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06482433-12a3-47f1-b34c-b3717137e81a None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:34.521310 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06482433-12a3-47f1-b34c-b3717137e81a None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:34.521549 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06482433-12a3-47f1-b34c-b3717137e81a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:34.540033 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-06482433-12a3-47f1-b34c-b3717137e81a None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:59:34.541153 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-06482433-12a3-47f1-b34c-b3717137e81a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:59:34.600329 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4dcedeb-eea8-47d9-bdb0-0c0b8e14174b None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:34.600918 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:34.608982 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 258/504] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:59:34 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:59:34.609987 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06482433-12a3-47f1-b34c-b3717137e81a None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:34.633841 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7471d99d-d3c9-4588-b955-2db5bfb64866 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:34.634438 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7471d99d-d3c9-4588-b955-2db5bfb64866 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:34.634713 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7471d99d-d3c9-4588-b955-2db5bfb64866 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:34.635010 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7471d99d-d3c9-4588-b955-2db5bfb64866 None admin] PATH_INFO: `/v3/domains` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:34.635639 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7471d99d-d3c9-4588-b955-2db5bfb64866 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:34.635936 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7471d99d-d3c9-4588-b955-2db5bfb64866 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:34.637292 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7471d99d-d3c9-4588-b955-2db5bfb64866 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:34.648956 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 247/505] 10.222.0.77 () {56 vars in 1109 bytes} [Tue Aug 30 04:59:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:34.649800 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7471d99d-d3c9-4588-b955-2db5bfb64866 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:34.705767 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:34.730180 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06482433-12a3-47f1-b34c-b3717137e81a None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:34.751077 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7471d99d-d3c9-4588-b955-2db5bfb64866 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:34.766301 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c47b30d-a211-43ad-8acd-f57d35377fe0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:34.767053 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c47b30d-a211-43ad-8acd-f57d35377fe0 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:34.767328 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c47b30d-a211-43ad-8acd-f57d35377fe0 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:34.767716 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c47b30d-a211-43ad-8acd-f57d35377fe0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:34.779091 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c47b30d-a211-43ad-8acd-f57d35377fe0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:34.780638 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c47b30d-a211-43ad-8acd-f57d35377fe0 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:34.784042 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8c9e45c-85a6-4246-ad91-5868542e3eb3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:34.784633 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8c9e45c-85a6-4246-ad91-5868542e3eb3 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:34.784908 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8c9e45c-85a6-4246-ad91-5868542e3eb3 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:34.785176 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8c9e45c-85a6-4246-ad91-5868542e3eb3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:34.795512 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-5c47b30d-a211-43ad-8acd-f57d35377fe0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:59:34.795512 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:34.795512 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:34.795512 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:34.795512 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:34.795512 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:34.795512 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:34.795512 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:34.795512 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:34.795512 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:34.795512 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:34.795512 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:34.795512 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:59:34.795512 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:59:34.795512 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:59:34.795512 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:59:34.796446 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:34.796446 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:34.796446 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:34.796446 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:34.796446 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:34.796446 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:34.796446 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:34.796446 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:34.796446 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:34.796446 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:34.796446 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:34.796446 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:34.796446 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:34.796446 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:34.796446 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:34.796446 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:34.796446 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:59:34.797292 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:59:34.797292 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:59:34.797292 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 04:59:34.797292 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 259/506] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 04:59:34 2022] GET /identity/v3/domains/Default => generated 87 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:34.797919 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c47b30d-a211-43ad-8acd-f57d35377fe0 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:34.848220 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:34.861178 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8c9e45c-85a6-4246-ad91-5868542e3eb3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:34.862650 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8c9e45c-85a6-4246-ad91-5868542e3eb3 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:34.902869 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c47b30d-a211-43ad-8acd-f57d35377fe0 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:34.916942 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:34.937208 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-517297b5-a7a5-469d-bb73-eba1c7b0bfdf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:34.937797 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-517297b5-a7a5-469d-bb73-eba1c7b0bfdf None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:34.938070 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-517297b5-a7a5-469d-bb73-eba1c7b0bfdf None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:34.938335 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-517297b5-a7a5-469d-bb73-eba1c7b0bfdf None admin] PATH_INFO: `/v3/domains` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:34.938991 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-517297b5-a7a5-469d-bb73-eba1c7b0bfdf None admin] RBAC: Adding query filter params (name=Default) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:34.939308 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-517297b5-a7a5-469d-bb73-eba1c7b0bfdf None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:34.940702 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-517297b5-a7a5-469d-bb73-eba1c7b0bfdf None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:34.952505 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 260/507] 10.222.0.77 () {56 vars in 1109 bytes} [Tue Aug 30 04:59:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:34.953279 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-517297b5-a7a5-469d-bb73-eba1c7b0bfdf None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:34.970842 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 248/508] 10.222.0.77 () {58 vars in 1282 bytes} [Tue Aug 30 04:59:34 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 317 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 04:59:34.970842 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8c9e45c-85a6-4246-ad91-5868542e3eb3 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:35.053910 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-517297b5-a7a5-469d-bb73-eba1c7b0bfdf None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:35.063312 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:35.064099 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8c9e45c-85a6-4246-ad91-5868542e3eb3 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:35.088195 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-572a7ef4-060b-4c57-bedd-3ceeccae083d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:35.088811 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-572a7ef4-060b-4c57-bedd-3ceeccae083d None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:35.089086 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-572a7ef4-060b-4c57-bedd-3ceeccae083d None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:35.089352 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-572a7ef4-060b-4c57-bedd-3ceeccae083d None admin] PATH_INFO: `/v3/users/nova` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:35.096246 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-572a7ef4-060b-4c57-bedd-3ceeccae083d None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:35.097738 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-572a7ef4-060b-4c57-bedd-3ceeccae083d None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:35.098745 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1e477e8-93ea-4dc9-be9e-b6396f001564 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:35.099374 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e477e8-93ea-4dc9-be9e-b6396f001564 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:35.099652 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e477e8-93ea-4dc9-be9e-b6396f001564 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:35.099918 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e477e8-93ea-4dc9-be9e-b6396f001564 None admin] PATH_INFO: `/v3/domains` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:35.100552 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1e477e8-93ea-4dc9-be9e-b6396f001564 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:35.100851 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1e477e8-93ea-4dc9-be9e-b6396f001564 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:35.102235 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1e477e8-93ea-4dc9-be9e-b6396f001564 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:35.110907 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-572a7ef4-060b-4c57-bedd-3ceeccae083d None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 30 04:59:35.110907 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:35.110907 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:35.110907 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:35.110907 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:35.110907 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:35.110907 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:35.110907 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:35.110907 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:35.110907 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:35.110907 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:35.110907 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:35.110907 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 04:59:35.110907 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 04:59:35.110907 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 04:59:35.110907 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 04:59:35.112067 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:35.112067 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:35.112067 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 04:59:35.112067 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:59:35.112067 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 04:59:35.112067 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:59:35.112067 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:35.112067 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:35.112067 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:35.112067 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:35.112067 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:35.112067 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:35.112067 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:35.112067 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:35.112067 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:35.112067 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:35.112067 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:35.112067 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:35.113191 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:35.113191 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:35.113191 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 04:59:35.113191 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 04:59:35.113191 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 04:59:35.113191 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 04:59:35.113191 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 04:59:35.113191 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 04:59:35.113191 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 30 04:59:35.113191 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 04:59:35.113191 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 261/509] 10.222.0.77 () {56 vars in 1093 bytes} [Tue Aug 30 04:59:34 2022] GET /identity/v3/users/nova => generated 82 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:35.113191 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-572a7ef4-060b-4c57-bedd-3ceeccae083d None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:35.117573 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 249/510] 10.222.0.77 () {56 vars in 1109 bytes} [Tue Aug 30 04:59:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:35.118627 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1e477e8-93ea-4dc9-be9e-b6396f001564 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:35.134264 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:35.213498 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-572a7ef4-060b-4c57-bedd-3ceeccae083d None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:35.215778 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1e477e8-93ea-4dc9-be9e-b6396f001564 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:35.247373 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4508823-b0c4-4c59-a2fe-0f13eed4a80f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:35.247986 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4508823-b0c4-4c59-a2fe-0f13eed4a80f None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:35.248264 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4508823-b0c4-4c59-a2fe-0f13eed4a80f None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:35.248535 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4508823-b0c4-4c59-a2fe-0f13eed4a80f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:35.249056 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfa7ec35-b3ca-4273-b994-27697c33657f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:35.249666 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa7ec35-b3ca-4273-b994-27697c33657f None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:35.249942 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa7ec35-b3ca-4273-b994-27697c33657f None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:35.250213 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa7ec35-b3ca-4273-b994-27697c33657f None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:35.257375 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfa7ec35-b3ca-4273-b994-27697c33657f None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:35.258861 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfa7ec35-b3ca-4273-b994-27697c33657f None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:35.258966 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4508823-b0c4-4c59-a2fe-0f13eed4a80f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:35.260399 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4508823-b0c4-4c59-a2fe-0f13eed4a80f None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:35.269758 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-bfa7ec35-b3ca-4273-b994-27697c33657f None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Aug 30 04:59:35.269758 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:35.269758 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:35.269758 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:35.269758 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:35.269758 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:35.269758 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:35.269758 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:35.269758 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:35.269758 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:35.269758 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:35.269758 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:35.269758 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 04:59:35.269758 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 04:59:35.269758 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 04:59:35.269758 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 04:59:35.270793 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:35.270793 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:35.270793 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 04:59:35.270793 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:59:35.270793 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 04:59:35.270793 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:59:35.270793 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:35.270793 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:35.270793 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:35.270793 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:35.270793 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:35.270793 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:35.270793 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:35.270793 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:35.270793 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:35.270793 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:35.270793 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:35.270793 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:35.272082 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:35.272082 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:35.272082 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 04:59:35.272082 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 04:59:35.272082 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 04:59:35.272082 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 04:59:35.272082 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 04:59:35.272082 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 04:59:35.272082 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Aug 30 04:59:35.272082 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 04:59:35.272082 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 250/511] 10.222.0.77 () {56 vars in 1099 bytes} [Tue Aug 30 04:59:35 2022] GET /identity/v3/users/cinder => generated 84 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:35.272082 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfa7ec35-b3ca-4273-b994-27697c33657f None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:35.277051 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-f4508823-b0c4-4c59-a2fe-0f13eed4a80f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:59:35.277051 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:35.277051 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:35.277051 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:35.277051 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:35.277051 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:35.277051 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:35.277051 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:35.277051 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:35.277051 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:35.277051 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:35.277051 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:35.277051 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:59:35.277051 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:59:35.277051 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:59:35.277051 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:59:35.278116 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:35.278116 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:35.278116 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:35.278116 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:35.278116 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:35.278116 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:35.278116 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:35.278116 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:35.278116 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:35.278116 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:35.278116 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:35.278116 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:35.278116 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:35.278116 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:35.278116 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:35.278116 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:35.278116 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:59:35.280420 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:59:35.280420 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:59:35.280420 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 04:59:35.280420 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 262/512] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 04:59:35 2022] GET /identity/v3/domains/Default => generated 87 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:35.280420 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4508823-b0c4-4c59-a2fe-0f13eed4a80f None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:35.303002 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:35.346558 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:35.372178 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfa7ec35-b3ca-4273-b994-27697c33657f None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:35.381892 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4508823-b0c4-4c59-a2fe-0f13eed4a80f None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:35.405228 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c2964d-a0d0-4b45-a331-7bd2a2b99c21 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:35.405845 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c2964d-a0d0-4b45-a331-7bd2a2b99c21 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:35.406128 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c2964d-a0d0-4b45-a331-7bd2a2b99c21 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:35.406409 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c2964d-a0d0-4b45-a331-7bd2a2b99c21 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:35.415203 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1baf40f-d791-40db-866a-7e65467bf5f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:35.415795 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1baf40f-d791-40db-866a-7e65467bf5f0 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:35.416073 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1baf40f-d791-40db-866a-7e65467bf5f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:35.416343 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1baf40f-d791-40db-866a-7e65467bf5f0 None admin] PATH_INFO: `/v3/users` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:35.417057 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1baf40f-d791-40db-866a-7e65467bf5f0 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:35.417274 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46c2964d-a0d0-4b45-a331-7bd2a2b99c21 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:35.417380 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1baf40f-d791-40db-866a-7e65467bf5f0 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:35.418747 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46c2964d-a0d0-4b45-a331-7bd2a2b99c21 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:35.418917 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1baf40f-d791-40db-866a-7e65467bf5f0 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:35.433424 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-46c2964d-a0d0-4b45-a331-7bd2a2b99c21 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:59:35.433424 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:35.433424 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:35.433424 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:35.433424 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:35.433424 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:35.433424 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:35.433424 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:35.433424 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:35.433424 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:35.433424 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:35.433424 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:35.433424 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:59:35.433424 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:59:35.433424 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:59:35.433424 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:59:35.434452 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:35.434452 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:35.434452 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:35.434452 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:35.434452 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:35.434452 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:35.434452 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:35.434452 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:35.434452 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:35.434452 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:35.434452 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:35.434452 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:35.434452 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:35.434452 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:35.434452 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:35.434452 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:35.434452 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:59:35.440409 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:59:35.440409 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:59:35.440409 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 04:59:35.440409 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 251/513] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 04:59:35 2022] GET /identity/v3/domains/Default => generated 87 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:35.440409 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c2964d-a0d0-4b45-a331-7bd2a2b99c21 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:35.454967 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 263/514] 10.222.0.77 () {56 vars in 1133 bytes} [Tue Aug 30 04:59:35 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:35.455802 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1baf40f-d791-40db-866a-7e65467bf5f0 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:35.532343 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:35.544963 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c2964d-a0d0-4b45-a331-7bd2a2b99c21 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:35.553876 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1baf40f-d791-40db-866a-7e65467bf5f0 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:35.569750 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:35.579143 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f59a6495-bf26-4c92-86ca-720acccaba57 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:35.579760 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f59a6495-bf26-4c92-86ca-720acccaba57 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:35.580042 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f59a6495-bf26-4c92-86ca-720acccaba57 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:35.580324 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f59a6495-bf26-4c92-86ca-720acccaba57 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:35.586690 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7afb6c7c-cec3-45c9-a676-16b5cf0c5381 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:35.587300 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7afb6c7c-cec3-45c9-a676-16b5cf0c5381 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:35.587572 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7afb6c7c-cec3-45c9-a676-16b5cf0c5381 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:35.587831 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7afb6c7c-cec3-45c9-a676-16b5cf0c5381 None admin] PATH_INFO: `/v3/users` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:35.588108 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f59a6495-bf26-4c92-86ca-720acccaba57 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:35.588532 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7afb6c7c-cec3-45c9-a676-16b5cf0c5381 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:35.588824 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7afb6c7c-cec3-45c9-a676-16b5cf0c5381 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:35.589559 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f59a6495-bf26-4c92-86ca-720acccaba57 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:35.590226 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7afb6c7c-cec3-45c9-a676-16b5cf0c5381 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:35.600608 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-f59a6495-bf26-4c92-86ca-720acccaba57 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Aug 30 04:59:35.600608 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:35.600608 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:35.600608 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:35.600608 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:35.600608 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:35.600608 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:35.600608 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:35.600608 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:35.600608 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:35.600608 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:35.600608 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:35.600608 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 04:59:35.600608 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 04:59:35.600608 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 04:59:35.600608 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 04:59:35.602221 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:35.602221 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:35.602221 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 04:59:35.602221 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:59:35.602221 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 04:59:35.602221 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:59:35.602221 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:35.602221 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:35.602221 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:35.602221 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:35.602221 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:35.602221 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:35.602221 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:35.602221 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:35.602221 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:35.602221 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:35.602221 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:35.602221 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:35.608160 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:35.608160 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:35.608160 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 04:59:35.608160 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 04:59:35.608160 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 04:59:35.608160 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 04:59:35.608160 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 04:59:35.608160 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 04:59:35.608160 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Aug 30 04:59:35.608160 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 04:59:35.608160 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 252/515] 10.222.0.77 () {56 vars in 1102 bytes} [Tue Aug 30 04:59:35 2022] GET /identity/v3/users/neutron => generated 85 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:35.608160 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f59a6495-bf26-4c92-86ca-720acccaba57 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:35.612430 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 264/516] 10.222.0.77 () {56 vars in 1137 bytes} [Tue Aug 30 04:59:35 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:35.613231 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7afb6c7c-cec3-45c9-a676-16b5cf0c5381 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:35.701193 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f59a6495-bf26-4c92-86ca-720acccaba57 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:35.710538 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7afb6c7c-cec3-45c9-a676-16b5cf0c5381 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:35.734638 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5cd7fb2-cbe5-4874-9533-0194d72abd22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:35.735256 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5cd7fb2-cbe5-4874-9533-0194d72abd22 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:35.735537 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5cd7fb2-cbe5-4874-9533-0194d72abd22 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:35.735807 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5cd7fb2-cbe5-4874-9533-0194d72abd22 None admin] PATH_INFO: `/v3/domains` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:35.736435 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5cd7fb2-cbe5-4874-9533-0194d72abd22 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:35.736736 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5cd7fb2-cbe5-4874-9533-0194d72abd22 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:35.738102 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5cd7fb2-cbe5-4874-9533-0194d72abd22 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:35.744312 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-915c9fba-2c06-4dbc-8602-ee71a1f48ea6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:35.744901 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-915c9fba-2c06-4dbc-8602-ee71a1f48ea6 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:35.745177 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-915c9fba-2c06-4dbc-8602-ee71a1f48ea6 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:35.745442 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-915c9fba-2c06-4dbc-8602-ee71a1f48ea6 None admin] PATH_INFO: `/v3/domains` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:35.746064 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-915c9fba-2c06-4dbc-8602-ee71a1f48ea6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:35.746358 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-915c9fba-2c06-4dbc-8602-ee71a1f48ea6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:35.747740 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-915c9fba-2c06-4dbc-8602-ee71a1f48ea6 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:35.749452 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 253/517] 10.222.0.77 () {56 vars in 1109 bytes} [Tue Aug 30 04:59:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:35.749966 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:35.750230 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5cd7fb2-cbe5-4874-9533-0194d72abd22 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:35.764066 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 265/518] 10.222.0.77 () {56 vars in 1109 bytes} [Tue Aug 30 04:59:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:35.765111 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-915c9fba-2c06-4dbc-8602-ee71a1f48ea6 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:35.796153 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:35.848915 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5cd7fb2-cbe5-4874-9533-0194d72abd22 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:35.867377 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-915c9fba-2c06-4dbc-8602-ee71a1f48ea6 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:35.882177 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50268032-b7e5-48f8-9f7b-d6bf76829a90 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:35.882801 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50268032-b7e5-48f8-9f7b-d6bf76829a90 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:35.883107 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50268032-b7e5-48f8-9f7b-d6bf76829a90 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:35.883381 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50268032-b7e5-48f8-9f7b-d6bf76829a90 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:35.901432 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6cb15ce-19a1-467f-b54d-76869104f659 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:35.902023 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6cb15ce-19a1-467f-b54d-76869104f659 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:35.902301 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6cb15ce-19a1-467f-b54d-76869104f659 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:35.902570 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6cb15ce-19a1-467f-b54d-76869104f659 None admin] PATH_INFO: `/v3/users` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:35.903315 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6cb15ce-19a1-467f-b54d-76869104f659 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:35.903619 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6cb15ce-19a1-467f-b54d-76869104f659 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:35.905045 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6cb15ce-19a1-467f-b54d-76869104f659 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:35.922257 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 266/519] 10.222.0.77 () {56 vars in 1139 bytes} [Tue Aug 30 04:59:35 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:35.923154 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6cb15ce-19a1-467f-b54d-76869104f659 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:35.950381 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50268032-b7e5-48f8-9f7b-d6bf76829a90 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:35.951848 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50268032-b7e5-48f8-9f7b-d6bf76829a90 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:35.979584 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:35.997361 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:36.024921 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6cb15ce-19a1-467f-b54d-76869104f659 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:36.048885 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 254/520] 10.222.0.77 () {58 vars in 1282 bytes} [Tue Aug 30 04:59:35 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 299 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 04:59:36.049663 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50268032-b7e5-48f8-9f7b-d6bf76829a90 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:36.058807 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0def338-1355-4913-8c5e-bda3cc7cd669 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:36.059434 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0def338-1355-4913-8c5e-bda3cc7cd669 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:36.059714 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0def338-1355-4913-8c5e-bda3cc7cd669 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:36.059986 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0def338-1355-4913-8c5e-bda3cc7cd669 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:36.126534 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0def338-1355-4913-8c5e-bda3cc7cd669 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:36.127955 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0def338-1355-4913-8c5e-bda3cc7cd669 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:36.147113 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50268032-b7e5-48f8-9f7b-d6bf76829a90 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:36.180820 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bd51367-61f2-417a-8f4e-289bce186040 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:36.181412 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bd51367-61f2-417a-8f4e-289bce186040 None admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:36.181541 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:36.181686 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bd51367-61f2-417a-8f4e-289bce186040 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:36.181953 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bd51367-61f2-417a-8f4e-289bce186040 None admin] PATH_INFO: `/v3/users` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:36.182590 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bd51367-61f2-417a-8f4e-289bce186040 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:36.184048 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bd51367-61f2-417a-8f4e-289bce186040 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:36.205541 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 255/521] 10.222.0.77 () {60 vars in 1132 bytes} [Tue Aug 30 04:59:36 2022] POST /identity/v3/users => generated 247 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:59:36.206577 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bd51367-61f2-417a-8f4e-289bce186040 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:36.208997 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:36.238992 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 267/522] 10.222.0.77 () {58 vars in 1282 bytes} [Tue Aug 30 04:59:35 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 316 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 04:59:36.240009 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0def338-1355-4913-8c5e-bda3cc7cd669 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:36.325914 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bd51367-61f2-417a-8f4e-289bce186040 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:36.352233 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0def338-1355-4913-8c5e-bda3cc7cd669 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:36.362293 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6192e1dd-433e-468c-984b-62e162558bf4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:36.362969 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6192e1dd-433e-468c-984b-62e162558bf4 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:36.363255 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6192e1dd-433e-468c-984b-62e162558bf4 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:36.363530 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6192e1dd-433e-468c-984b-62e162558bf4 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:36.371926 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6192e1dd-433e-468c-984b-62e162558bf4 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:36.373394 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6192e1dd-433e-468c-984b-62e162558bf4 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:36.387489 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-6192e1dd-433e-468c-984b-62e162558bf4 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:59:36.387489 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:36.387489 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:36.387489 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:36.387489 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:36.387489 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:36.387489 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:36.387489 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:36.387489 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:36.387489 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:36.387489 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:36.387489 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:36.387489 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 04:59:36.387489 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 04:59:36.387489 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 04:59:36.388534 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 04:59:36.388534 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:36.388534 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:36.388534 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:36.388534 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:36.388534 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:36.388534 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:36.388534 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:36.388534 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:36.388534 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:36.388534 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:36.388534 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:36.388534 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:36.388534 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:36.388534 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:36.388534 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:36.388534 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:36.389682 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 04:59:36.389682 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 04:59:36.389682 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 04:59:36.389682 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 04:59:36.389682 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 04:59:36.389682 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 04:59:36.389682 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:59:36.389682 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 04:59:36.389682 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 256/523] 10.222.0.77 () {56 vars in 1111 bytes} [Tue Aug 30 04:59:36 2022] GET /identity/v3/projects/service => generated 88 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:36.389682 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6192e1dd-433e-468c-984b-62e162558bf4 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:36.399572 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c894164-ee01-45b8-88d3-f4d48f3325f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:36.400186 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c894164-ee01-45b8-88d3-f4d48f3325f8 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:36.400511 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c894164-ee01-45b8-88d3-f4d48f3325f8 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:36.400784 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c894164-ee01-45b8-88d3-f4d48f3325f8 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:36.426304 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:36.460701 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:36.483819 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c894164-ee01-45b8-88d3-f4d48f3325f8 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:36.485297 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c894164-ee01-45b8-88d3-f4d48f3325f8 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:36.504351 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6192e1dd-433e-468c-984b-62e162558bf4 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:36.538866 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-286cb162-e40e-4cc2-8c26-0b5f4526594a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:36.539756 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-286cb162-e40e-4cc2-8c26-0b5f4526594a None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:36.540096 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-286cb162-e40e-4cc2-8c26-0b5f4526594a None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:36.540309 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-286cb162-e40e-4cc2-8c26-0b5f4526594a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:36.551450 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-286cb162-e40e-4cc2-8c26-0b5f4526594a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:36.552893 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-286cb162-e40e-4cc2-8c26-0b5f4526594a None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:36.567721 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-286cb162-e40e-4cc2-8c26-0b5f4526594a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:59:36.567721 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:36.567721 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:36.567721 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:36.567721 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:36.567721 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:36.567721 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:36.567721 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:36.567721 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:36.567721 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:36.567721 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:36.567721 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:36.567721 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:59:36.567721 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:59:36.567721 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:59:36.567721 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:59:36.568637 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:36.568637 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:36.568637 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:36.568637 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:36.568637 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:36.568637 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:36.568637 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:36.568637 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:36.568637 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:36.568637 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:36.568637 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:36.568637 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:36.568637 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:36.568637 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:36.568637 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:36.568637 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:36.568637 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:59:36.569497 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:59:36.569497 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:59:36.569497 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 04:59:36.569497 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 257/524] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 04:59:36 2022] GET /identity/v3/domains/Default => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:36.569708 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-286cb162-e40e-4cc2-8c26-0b5f4526594a None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:36.605553 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 268/525] 10.222.0.77 () {58 vars in 1282 bytes} [Tue Aug 30 04:59:36 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 361 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 04:59:36.605553 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c894164-ee01-45b8-88d3-f4d48f3325f8 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:36.661564 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:36.680633 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-286cb162-e40e-4cc2-8c26-0b5f4526594a None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:36.695725 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:36.715701 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c894164-ee01-45b8-88d3-f4d48f3325f8 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:36.717421 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f835a194-61a1-4860-b7ce-fddee021f005 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:36.718050 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f835a194-61a1-4860-b7ce-fddee021f005 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:36.718331 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f835a194-61a1-4860-b7ce-fddee021f005 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:36.718607 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f835a194-61a1-4860-b7ce-fddee021f005 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:36.730405 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f835a194-61a1-4860-b7ce-fddee021f005 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:36.731916 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f835a194-61a1-4860-b7ce-fddee021f005 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:36.746768 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-f835a194-61a1-4860-b7ce-fddee021f005 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:59:36.746768 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:36.746768 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:36.746768 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:36.746768 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:36.746768 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:36.746768 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:36.746768 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:36.746768 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:36.746768 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:36.746768 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:36.746768 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:36.746768 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:59:36.746768 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:59:36.746768 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:59:36.746768 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:59:36.748053 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:36.748053 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:36.748053 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:36.748053 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:36.748053 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:36.748053 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:36.748053 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:36.748053 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:36.748053 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:36.748053 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:36.748053 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:36.748053 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:36.748053 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:36.748053 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:36.748053 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:36.748053 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:36.748053 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:59:36.748950 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:59:36.748950 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:59:36.748950 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 04:59:36.748950 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 258/526] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 04:59:36 2022] GET /identity/v3/domains/Default => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:36.748950 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f835a194-61a1-4860-b7ce-fddee021f005 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:36.759425 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1df310e0-916a-4b07-8265-6cccf3e97bac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:36.760117 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1df310e0-916a-4b07-8265-6cccf3e97bac None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:36.760398 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1df310e0-916a-4b07-8265-6cccf3e97bac None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:36.760874 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1df310e0-916a-4b07-8265-6cccf3e97bac None admin] PATH_INFO: `/v3/projects` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:36.761375 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1df310e0-916a-4b07-8265-6cccf3e97bac None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:36.761651 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1df310e0-916a-4b07-8265-6cccf3e97bac None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:36.763123 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1df310e0-916a-4b07-8265-6cccf3e97bac None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:36.786657 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 269/527] 10.222.0.77 () {56 vars in 1148 bytes} [Tue Aug 30 04:59:36 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:36.786657 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1df310e0-916a-4b07-8265-6cccf3e97bac None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:36.855106 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f835a194-61a1-4860-b7ce-fddee021f005 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:36.890308 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-980488ad-3e11-4049-8425-bbb4d1b50a66 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:36.890966 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-980488ad-3e11-4049-8425-bbb4d1b50a66 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:36.891260 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-980488ad-3e11-4049-8425-bbb4d1b50a66 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:36.891516 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-980488ad-3e11-4049-8425-bbb4d1b50a66 None admin] PATH_INFO: `/v3/domains` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:36.892157 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-980488ad-3e11-4049-8425-bbb4d1b50a66 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:36.892460 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-980488ad-3e11-4049-8425-bbb4d1b50a66 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:36.893845 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-980488ad-3e11-4049-8425-bbb4d1b50a66 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:36.897738 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1df310e0-916a-4b07-8265-6cccf3e97bac None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:36.906111 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 259/528] 10.222.0.77 () {56 vars in 1109 bytes} [Tue Aug 30 04:59:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:36.906937 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-980488ad-3e11-4049-8425-bbb4d1b50a66 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:36.917343 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:36.937582 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:36.940469 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6810ecac-1852-479d-b31a-67917fce6df1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:36.941123 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6810ecac-1852-479d-b31a-67917fce6df1 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:36.941398 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6810ecac-1852-479d-b31a-67917fce6df1 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:36.941679 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6810ecac-1852-479d-b31a-67917fce6df1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:36.951801 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6810ecac-1852-479d-b31a-67917fce6df1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:36.953254 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6810ecac-1852-479d-b31a-67917fce6df1 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:36.968967 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-6810ecac-1852-479d-b31a-67917fce6df1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:59:36.968967 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:36.968967 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:36.968967 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:36.968967 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:36.968967 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:36.968967 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:36.968967 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:36.968967 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:36.968967 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:36.968967 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:36.968967 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:36.968967 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:59:36.968967 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:59:36.968967 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:59:36.968967 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:59:36.970059 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:36.970059 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:36.970059 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:36.970059 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:36.970059 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:36.970059 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:36.970059 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:36.970059 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:36.970059 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:36.970059 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:36.970059 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:36.970059 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:36.970059 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:36.970059 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:36.970059 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:36.970059 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:36.970059 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:59:36.975565 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:59:36.975565 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:59:36.975565 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 04:59:36.975565 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 270/529] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 04:59:36 2022] GET /identity/v3/domains/Default => generated 87 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:36.975565 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6810ecac-1852-479d-b31a-67917fce6df1 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:37.021752 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-980488ad-3e11-4049-8425-bbb4d1b50a66 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:37.057742 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf3e19f0-2eaf-41a3-ba65-7955b4b82710 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:37.058475 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf3e19f0-2eaf-41a3-ba65-7955b4b82710 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:37.058909 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf3e19f0-2eaf-41a3-ba65-7955b4b82710 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:37.059286 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf3e19f0-2eaf-41a3-ba65-7955b4b82710 None admin] PATH_INFO: `/v3/domains` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:37.060025 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf3e19f0-2eaf-41a3-ba65-7955b4b82710 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:37.060335 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf3e19f0-2eaf-41a3-ba65-7955b4b82710 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:37.061699 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf3e19f0-2eaf-41a3-ba65-7955b4b82710 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:37.076958 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 260/530] 10.222.0.77 () {56 vars in 1109 bytes} [Tue Aug 30 04:59:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:37.076958 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf3e19f0-2eaf-41a3-ba65-7955b4b82710 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:37.082203 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6810ecac-1852-479d-b31a-67917fce6df1 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:37.123445 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9056c882-b41d-4a61-bf44-4a7241bbfd9d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:37.124048 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9056c882-b41d-4a61-bf44-4a7241bbfd9d None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:37.124326 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9056c882-b41d-4a61-bf44-4a7241bbfd9d None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:37.124600 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9056c882-b41d-4a61-bf44-4a7241bbfd9d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:37.125321 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9056c882-b41d-4a61-bf44-4a7241bbfd9d None admin] RBAC: Adding query filter params (role.id=821647e8b40542499667e29e1b301cb2, scope.project.id=e9919141e3b947ab8d91b55bba4a7de6, user.id=e0a002ac0bf240a891ac7304178d3f0e) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:37.125625 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9056c882-b41d-4a61-bf44-4a7241bbfd9d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:37.127092 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9056c882-b41d-4a61-bf44-4a7241bbfd9d None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:37.142858 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 271/531] 10.222.0.77 () {56 vars in 1374 bytes} [Tue Aug 30 04:59:36 2022] GET /identity/v3/role_assignments?user.id=e0a002ac0bf240a891ac7304178d3f0e&scope.project.id=e9919141e3b947ab8d91b55bba4a7de6&role.id=821647e8b40542499667e29e1b301cb2 => generated 259 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:37.142858 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9056c882-b41d-4a61-bf44-4a7241bbfd9d None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:37.171665 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:37.199554 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:37.220948 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf3e19f0-2eaf-41a3-ba65-7955b4b82710 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:37.256402 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9056c882-b41d-4a61-bf44-4a7241bbfd9d None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:37.267332 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2d507fe-e573-481d-8820-eac04020a12d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:37.269446 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d507fe-e573-481d-8820-eac04020a12d None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:37.270526 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d507fe-e573-481d-8820-eac04020a12d None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:37.270927 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d507fe-e573-481d-8820-eac04020a12d None admin] PATH_INFO: `/v3/projects/service` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:37.278949 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2d507fe-e573-481d-8820-eac04020a12d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:37.280543 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2d507fe-e573-481d-8820-eac04020a12d None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:37.293147 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-b2d507fe-e573-481d-8820-eac04020a12d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:59:37.293147 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:37.293147 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:37.293147 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:37.293147 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:37.293147 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:37.293147 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:37.293147 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:37.293147 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:37.293147 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:37.293147 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:37.293147 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:37.293147 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 04:59:37.293147 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 04:59:37.293147 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 04:59:37.294206 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 04:59:37.294206 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:37.294206 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:37.294206 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:37.294206 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:37.294206 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:37.294206 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:37.294206 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:37.294206 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:37.294206 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:37.294206 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:37.294206 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:37.294206 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:37.294206 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:37.294206 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:37.294206 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:37.294206 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:37.295267 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 04:59:37.295267 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 04:59:37.295267 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 04:59:37.295267 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 04:59:37.295267 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 04:59:37.295267 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 04:59:37.295267 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:59:37.295267 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 04:59:37.295267 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 261/532] 10.222.0.77 () {56 vars in 1111 bytes} [Tue Aug 30 04:59:37 2022] GET /identity/v3/projects/service => generated 88 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:37.295762 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2d507fe-e573-481d-8820-eac04020a12d None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:37.302940 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a080a24-6450-4f59-b124-78a51c5c1bdb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:37.303749 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a080a24-6450-4f59-b124-78a51c5c1bdb None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:37.304171 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a080a24-6450-4f59-b124-78a51c5c1bdb None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:37.304619 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a080a24-6450-4f59-b124-78a51c5c1bdb None admin] PATH_INFO: `/v3/domains` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:37.305829 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a080a24-6450-4f59-b124-78a51c5c1bdb None admin] RBAC: Adding query filter params (name=Default) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:37.306274 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a080a24-6450-4f59-b124-78a51c5c1bdb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:37.308521 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a080a24-6450-4f59-b124-78a51c5c1bdb None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:37.325972 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 272/533] 10.222.0.77 () {56 vars in 1109 bytes} [Tue Aug 30 04:59:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:37.327397 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a080a24-6450-4f59-b124-78a51c5c1bdb None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:37.406421 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2d507fe-e573-481d-8820-eac04020a12d None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:37.442229 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-299bb9f8-8672-4ba7-8ffc-0e5dd63b99f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:37.442768 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-299bb9f8-8672-4ba7-8ffc-0e5dd63b99f4 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:37.443137 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-299bb9f8-8672-4ba7-8ffc-0e5dd63b99f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:37.443410 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-299bb9f8-8672-4ba7-8ffc-0e5dd63b99f4 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:37.450991 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-299bb9f8-8672-4ba7-8ffc-0e5dd63b99f4 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:37.452298 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:37.457677 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-299bb9f8-8672-4ba7-8ffc-0e5dd63b99f4 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:37.463834 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a080a24-6450-4f59-b124-78a51c5c1bdb None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:37.475719 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-299bb9f8-8672-4ba7-8ffc-0e5dd63b99f4 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:59:37.475719 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:37.475719 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:37.475719 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:37.475719 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:37.475719 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:37.475719 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:37.475719 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:37.475719 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:37.475719 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:37.475719 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:37.475719 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:37.475719 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 04:59:37.475719 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 04:59:37.475719 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 04:59:37.476844 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 04:59:37.476844 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:37.476844 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:37.476844 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:37.476844 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:37.476844 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:37.476844 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:37.476844 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:37.476844 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:37.476844 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:37.476844 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:37.476844 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:37.476844 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:37.476844 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:37.476844 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:37.476844 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:37.476844 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:37.477892 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 04:59:37.477892 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 04:59:37.477892 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 04:59:37.477892 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 04:59:37.477892 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 04:59:37.477892 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 04:59:37.477892 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:59:37.477892 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 04:59:37.479419 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 262/534] 10.222.0.77 () {56 vars in 1111 bytes} [Tue Aug 30 04:59:37 2022] GET /identity/v3/projects/service => generated 88 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:37.480467 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-299bb9f8-8672-4ba7-8ffc-0e5dd63b99f4 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:37.490235 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:37.511961 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c091b8a-8fbd-4540-9340-2c4f3d52c558 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:37.512577 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c091b8a-8fbd-4540-9340-2c4f3d52c558 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:37.512856 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c091b8a-8fbd-4540-9340-2c4f3d52c558 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:37.513128 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c091b8a-8fbd-4540-9340-2c4f3d52c558 None admin] PATH_INFO: `/v3/projects` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:37.514144 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c091b8a-8fbd-4540-9340-2c4f3d52c558 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:37.514594 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c091b8a-8fbd-4540-9340-2c4f3d52c558 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:37.516460 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c091b8a-8fbd-4540-9340-2c4f3d52c558 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:37.532101 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 273/535] 10.222.0.77 () {56 vars in 1148 bytes} [Tue Aug 30 04:59:37 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:37.534479 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c091b8a-8fbd-4540-9340-2c4f3d52c558 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:37.612856 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-299bb9f8-8672-4ba7-8ffc-0e5dd63b99f4 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:37.642096 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c091b8a-8fbd-4540-9340-2c4f3d52c558 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:37.661594 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5b815fa-6498-4352-bb68-08ad9cab28fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:37.662547 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b815fa-6498-4352-bb68-08ad9cab28fa None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:37.663098 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b815fa-6498-4352-bb68-08ad9cab28fa None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:37.664118 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b815fa-6498-4352-bb68-08ad9cab28fa None admin] PATH_INFO: `/v3/projects/service` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:37.674415 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5b815fa-6498-4352-bb68-08ad9cab28fa None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:37.676214 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5b815fa-6498-4352-bb68-08ad9cab28fa None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:37.678246 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c9ded01-2006-43cf-81fe-ae7e188477ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:37.678955 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c9ded01-2006-43cf-81fe-ae7e188477ce None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:37.679305 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c9ded01-2006-43cf-81fe-ae7e188477ce None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:37.679911 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c9ded01-2006-43cf-81fe-ae7e188477ce None admin] PATH_INFO: `/v3/projects` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:37.680424 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c9ded01-2006-43cf-81fe-ae7e188477ce None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:37.680792 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c9ded01-2006-43cf-81fe-ae7e188477ce None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:37.682349 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c9ded01-2006-43cf-81fe-ae7e188477ce None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:37.691247 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-a5b815fa-6498-4352-bb68-08ad9cab28fa None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:59:37.691247 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:37.691247 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:37.691247 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:37.691247 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:37.691247 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:37.691247 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:37.691247 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:37.691247 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:37.691247 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:37.691247 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:37.691247 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:37.691247 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 04:59:37.691247 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 04:59:37.691247 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 04:59:37.692085 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 04:59:37.692085 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:37.692085 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:37.692085 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:37.692085 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:37.692085 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:37.692085 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:37.692085 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:37.692085 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:37.692085 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:37.692085 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:37.692085 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:37.692085 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:37.692085 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:37.692085 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:37.692085 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:37.692085 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:37.692960 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 04:59:37.692960 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 04:59:37.692960 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 04:59:37.692960 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 04:59:37.692960 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 04:59:37.692960 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 04:59:37.692960 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:59:37.692960 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 04:59:37.702847 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 274/536] 10.222.0.77 () {56 vars in 1148 bytes} [Tue Aug 30 04:59:37 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:37.702847 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c9ded01-2006-43cf-81fe-ae7e188477ce None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:37.703048 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 263/537] 10.222.0.77 () {56 vars in 1111 bytes} [Tue Aug 30 04:59:37 2022] GET /identity/v3/projects/service => generated 88 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:37.706088 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5b815fa-6498-4352-bb68-08ad9cab28fa None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:37.718367 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:37.802715 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:37.820176 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c9ded01-2006-43cf-81fe-ae7e188477ce None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:37.854654 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5b815fa-6498-4352-bb68-08ad9cab28fa None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:37.858220 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-858d3da9-5069-415b-b49f-a1dafc14d301 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:37.858726 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-858d3da9-5069-415b-b49f-a1dafc14d301 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:37.859037 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-858d3da9-5069-415b-b49f-a1dafc14d301 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:37.859307 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-858d3da9-5069-415b-b49f-a1dafc14d301 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:37.860021 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-858d3da9-5069-415b-b49f-a1dafc14d301 None admin] RBAC: Adding query filter params (role.id=821647e8b40542499667e29e1b301cb2, scope.project.id=e9919141e3b947ab8d91b55bba4a7de6, user.id=886eb5e00bbf4ec292eb962b665b37d0) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:37.860326 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-858d3da9-5069-415b-b49f-a1dafc14d301 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:37.861751 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-858d3da9-5069-415b-b49f-a1dafc14d301 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:37.868671 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 275/538] 10.222.0.77 () {56 vars in 1374 bytes} [Tue Aug 30 04:59:37 2022] GET /identity/v3/role_assignments?user.id=886eb5e00bbf4ec292eb962b665b37d0&scope.project.id=e9919141e3b947ab8d91b55bba4a7de6&role.id=821647e8b40542499667e29e1b301cb2 => generated 259 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:37.869731 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-858d3da9-5069-415b-b49f-a1dafc14d301 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:37.910653 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46198883-9461-41cc-a9f3-a7aa6addb25f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:37.912789 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46198883-9461-41cc-a9f3-a7aa6addb25f None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:37.913069 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46198883-9461-41cc-a9f3-a7aa6addb25f None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:37.913340 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46198883-9461-41cc-a9f3-a7aa6addb25f None admin] PATH_INFO: `/v3/projects` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:37.915070 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46198883-9461-41cc-a9f3-a7aa6addb25f None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:37.915380 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46198883-9461-41cc-a9f3-a7aa6addb25f None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:37.916864 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46198883-9461-41cc-a9f3-a7aa6addb25f None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:37.961708 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 264/539] 10.222.0.77 () {56 vars in 1148 bytes} [Tue Aug 30 04:59:37 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 257 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:37.966023 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46198883-9461-41cc-a9f3-a7aa6addb25f None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:38.003412 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:38.046342 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-858d3da9-5069-415b-b49f-a1dafc14d301 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:38.084468 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45fd3f74-c4d2-47c3-b53f-4660816656e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:38.085108 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45fd3f74-c4d2-47c3-b53f-4660816656e2 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:38.085386 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45fd3f74-c4d2-47c3-b53f-4660816656e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:38.085699 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45fd3f74-c4d2-47c3-b53f-4660816656e2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:38.086425 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45fd3f74-c4d2-47c3-b53f-4660816656e2 None admin] RBAC: Adding query filter params (role.id=821647e8b40542499667e29e1b301cb2, scope.project.id=e9919141e3b947ab8d91b55bba4a7de6, user.id=019a79b2affe478b8a30d1c538d0415c) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:38.086731 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45fd3f74-c4d2-47c3-b53f-4660816656e2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:38.088286 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45fd3f74-c4d2-47c3-b53f-4660816656e2 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:38.104731 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 276/540] 10.222.0.77 () {56 vars in 1374 bytes} [Tue Aug 30 04:59:37 2022] GET /identity/v3/role_assignments?user.id=019a79b2affe478b8a30d1c538d0415c&scope.project.id=e9919141e3b947ab8d91b55bba4a7de6&role.id=821647e8b40542499667e29e1b301cb2 => generated 259 bytes in 235 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:38.125181 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46198883-9461-41cc-a9f3-a7aa6addb25f None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:38.156003 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:38.166613 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14292d94-a3f2-4fae-8425-83bc0ce6bc06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:38.167463 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14292d94-a3f2-4fae-8425-83bc0ce6bc06 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:38.167744 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14292d94-a3f2-4fae-8425-83bc0ce6bc06 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:38.168019 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14292d94-a3f2-4fae-8425-83bc0ce6bc06 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:38.168761 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14292d94-a3f2-4fae-8425-83bc0ce6bc06 None admin] RBAC: Adding query filter params (role.id=821647e8b40542499667e29e1b301cb2, scope.project.id=e9919141e3b947ab8d91b55bba4a7de6, user.id=c33c45f4770d4151858e1d897d09af69) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:38.169068 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14292d94-a3f2-4fae-8425-83bc0ce6bc06 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:38.170521 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14292d94-a3f2-4fae-8425-83bc0ce6bc06 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:38.176882 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 265/541] 10.222.0.77 () {56 vars in 1374 bytes} [Tue Aug 30 04:59:37 2022] GET /identity/v3/role_assignments?user.id=c33c45f4770d4151858e1d897d09af69&scope.project.id=e9919141e3b947ab8d91b55bba4a7de6&role.id=821647e8b40542499667e29e1b301cb2 => generated 259 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:38.301983 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-223ab7c7-cc6f-4774-8423-2419a836d76f None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:38.302334 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-223ab7c7-cc6f-4774-8423-2419a836d76f None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:38.302611 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-223ab7c7-cc6f-4774-8423-2419a836d76f None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:38.305032 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 277/542] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:59:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:59:38.318653 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e2c3a6d-5ade-4825-8db9-2947c217e27e None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:38.319018 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e2c3a6d-5ade-4825-8db9-2947c217e27e None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:38.319243 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e2c3a6d-5ade-4825-8db9-2947c217e27e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:38.344866 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-2e2c3a6d-5ade-4825-8db9-2947c217e27e None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:59:38.346108 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-2e2c3a6d-5ade-4825-8db9-2947c217e27e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:59:38.443274 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 266/543] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:59:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:59:38.835791 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-223ab7c7-cc6f-4774-8423-2419a836d76f None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:38.968467 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-223ab7c7-cc6f-4774-8423-2419a836d76f None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:39.014856 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1dc931f-8bb8-44f0-bb7b-e4a8a5599563 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:39.015755 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1dc931f-8bb8-44f0-bb7b-e4a8a5599563 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:39.016179 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1dc931f-8bb8-44f0-bb7b-e4a8a5599563 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:39.016590 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1dc931f-8bb8-44f0-bb7b-e4a8a5599563 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:39.030841 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:39.035584 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1dc931f-8bb8-44f0-bb7b-e4a8a5599563 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:59:39.041062 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-c1dc931f-8bb8-44f0-bb7b-e4a8a5599563 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 30 04:59:39.041062 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:39.041062 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:39.041062 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:39.041062 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:39.041062 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:39.041062 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:39.041062 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:39.041062 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:39.041062 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:39.041062 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:39.041062 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:39.041062 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:59:39.041062 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:59:39.041062 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:59:39.041062 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:59:39.042125 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:59:39.042125 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:59:39.042125 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:59:39.042125 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:59:39.042125 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:39.042125 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:39.042125 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:39.042125 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:39.042125 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:39.042125 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:39.042125 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:39.042125 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:39.042125 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:39.042125 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:39.042125 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:39.042125 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:39.043177 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:39.043177 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:39.043177 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:39.043177 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:39.043177 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:59:39.043177 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:59:39.043177 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:59:39.043177 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:59:39.043177 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:59:39.043177 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:59:39.043177 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 30 04:59:39.043177 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 04:59:39.045375 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 278/544] 10.222.0.77 () {56 vars in 1102 bytes} [Tue Aug 30 04:59:38 2022] GET /identity/v3/roles/service => generated 85 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:39.050904 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e2c3a6d-5ade-4825-8db9-2947c217e27e None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:39.165429 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e2c3a6d-5ade-4825-8db9-2947c217e27e None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:39.187132 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:39.206422 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1956307f-3642-478f-af6a-4535daceb549 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:39.207262 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1956307f-3642-478f-af6a-4535daceb549 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:39.207683 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1956307f-3642-478f-af6a-4535daceb549 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:39.208091 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1956307f-3642-478f-af6a-4535daceb549 None admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:39.208885 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1956307f-3642-478f-af6a-4535daceb549 None admin] RBAC: Adding query filter params (name=service) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:39.209348 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1956307f-3642-478f-af6a-4535daceb549 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:39.210898 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1956307f-3642-478f-af6a-4535daceb549 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:39.219766 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 267/545] 10.222.0.77 () {56 vars in 1103 bytes} [Tue Aug 30 04:59:39 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:39.224949 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1dc931f-8bb8-44f0-bb7b-e4a8a5599563 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:39.244604 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba595f4e-9aac-4198-9473-bb27d239de81 None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:39.244913 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba595f4e-9aac-4198-9473-bb27d239de81 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:39.245161 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba595f4e-9aac-4198-9473-bb27d239de81 None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:39.246993 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 268/546] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:59:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 04:59:39.251894 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bfd27cc-a71f-4353-bc5c-f1fad225e647 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:39.252173 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bfd27cc-a71f-4353-bc5c-f1fad225e647 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:39.252446 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bfd27cc-a71f-4353-bc5c-f1fad225e647 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:39.289948 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-1bfd27cc-a71f-4353-bc5c-f1fad225e647 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:59:39.291148 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-1bfd27cc-a71f-4353-bc5c-f1fad225e647 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:59:39.392200 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1dc931f-8bb8-44f0-bb7b-e4a8a5599563 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:39.445168 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 269/547] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:59:39 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:59:39.455661 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fd02342-676d-4b4e-828e-3a40bc2c7749 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:39.456293 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fd02342-676d-4b4e-828e-3a40bc2c7749 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:39.456571 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fd02342-676d-4b4e-828e-3a40bc2c7749 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:39.456838 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fd02342-676d-4b4e-828e-3a40bc2c7749 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:39.469603 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fd02342-676d-4b4e-828e-3a40bc2c7749 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:39.470661 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fd02342-676d-4b4e-828e-3a40bc2c7749 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:39.487944 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-3fd02342-676d-4b4e-828e-3a40bc2c7749 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:59:39.487944 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:39.487944 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:39.487944 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:39.487944 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:39.487944 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:39.487944 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:39.487944 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:39.487944 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:39.487944 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:39.487944 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:39.487944 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:39.487944 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:59:39.487944 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:59:39.487944 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:59:39.487944 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:59:39.488876 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:39.488876 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:39.488876 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:39.488876 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:39.488876 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:39.488876 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:39.488876 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:39.488876 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:39.488876 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:39.488876 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:39.488876 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:39.488876 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:39.488876 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:39.488876 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:39.488876 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:39.488876 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:39.488876 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:59:39.490302 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:59:39.490302 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:59:39.490302 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 04:59:39.490302 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 279/548] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 04:59:39 2022] GET /identity/v3/domains/Default => generated 87 bytes in 265 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:39.496150 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bfd27cc-a71f-4353-bc5c-f1fad225e647 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:39.657712 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bfd27cc-a71f-4353-bc5c-f1fad225e647 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:39.681009 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:39.708900 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7774970d-6deb-4f8c-9c46-1681dc17a609 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:39.709810 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7774970d-6deb-4f8c-9c46-1681dc17a609 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:39.710301 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7774970d-6deb-4f8c-9c46-1681dc17a609 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:39.711348 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7774970d-6deb-4f8c-9c46-1681dc17a609 None admin] PATH_INFO: `/v3/domains` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:39.712181 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7774970d-6deb-4f8c-9c46-1681dc17a609 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:39.712691 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7774970d-6deb-4f8c-9c46-1681dc17a609 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:39.714259 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7774970d-6deb-4f8c-9c46-1681dc17a609 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:39.729819 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 270/549] 10.222.0.77 () {56 vars in 1109 bytes} [Tue Aug 30 04:59:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 234 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:39.734941 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fd02342-676d-4b4e-828e-3a40bc2c7749 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:39.807683 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7774970d-6deb-4f8c-9c46-1681dc17a609 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:39.809054 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:39.870312 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fd02342-676d-4b4e-828e-3a40bc2c7749 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:39.903619 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46b1a1b7-218d-4d62-a2ae-a248e9ce3713 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:39.904255 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b1a1b7-218d-4d62-a2ae-a248e9ce3713 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:39.904688 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b1a1b7-218d-4d62-a2ae-a248e9ce3713 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:39.904964 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b1a1b7-218d-4d62-a2ae-a248e9ce3713 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:39.922216 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46b1a1b7-218d-4d62-a2ae-a248e9ce3713 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:39.923721 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46b1a1b7-218d-4d62-a2ae-a248e9ce3713 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:39.944654 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-46b1a1b7-218d-4d62-a2ae-a248e9ce3713 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Aug 30 04:59:39.944654 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:39.944654 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:39.944654 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:39.944654 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:39.944654 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:39.944654 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:39.944654 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:39.944654 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:39.944654 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:39.944654 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:39.944654 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:39.944654 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 04:59:39.944654 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 04:59:39.944654 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 04:59:39.944654 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 04:59:39.946685 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:39.946685 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:39.946685 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 04:59:39.946685 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:59:39.946685 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 04:59:39.946685 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:59:39.946685 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:39.946685 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:39.946685 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:39.946685 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:39.946685 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:39.946685 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:39.946685 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:39.946685 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:39.946685 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:39.946685 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:39.946685 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:39.946685 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:39.949263 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:39.949263 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:39.949263 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 04:59:39.949263 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 04:59:39.949263 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 04:59:39.949263 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 04:59:39.949263 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 04:59:39.949263 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 04:59:39.949263 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Aug 30 04:59:39.949263 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 04:59:39.949263 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 280/550] 10.222.0.77 () {56 vars in 1096 bytes} [Tue Aug 30 04:59:39 2022] GET /identity/v3/users/swift => generated 83 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:39.954849 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46b1a1b7-218d-4d62-a2ae-a248e9ce3713 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:40.038242 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7774970d-6deb-4f8c-9c46-1681dc17a609 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:40.109234 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e280471e-6bc0-4915-886a-c219160397fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:40.109857 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e280471e-6bc0-4915-886a-c219160397fd None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:40.110131 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e280471e-6bc0-4915-886a-c219160397fd None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:40.110402 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e280471e-6bc0-4915-886a-c219160397fd None admin] PATH_INFO: `/v3/roles/service` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:40.134519 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e280471e-6bc0-4915-886a-c219160397fd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:59:40.139940 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-e280471e-6bc0-4915-886a-c219160397fd None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 30 04:59:40.139940 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:40.139940 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:40.139940 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:40.139940 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:40.139940 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:40.139940 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:40.139940 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:40.139940 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:40.139940 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:40.139940 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:40.139940 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:40.139940 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:59:40.139940 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:59:40.139940 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:59:40.139940 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:59:40.140825 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:59:40.140825 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:59:40.140825 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:59:40.140825 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:59:40.140825 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:40.140825 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:40.140825 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:40.140825 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:40.140825 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:40.140825 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:40.140825 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:40.140825 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:40.140825 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:40.140825 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:40.140825 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:40.140825 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:40.141778 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:40.141778 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:40.141778 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:40.141778 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:40.141778 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:59:40.141778 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:59:40.141778 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:59:40.141778 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:59:40.141778 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:59:40.141778 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:59:40.141778 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 30 04:59:40.141778 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 04:59:40.142878 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46b1a1b7-218d-4d62-a2ae-a248e9ce3713 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:40.147259 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 271/551] 10.222.0.77 () {56 vars in 1102 bytes} [Tue Aug 30 04:59:39 2022] GET /identity/v3/roles/service => generated 85 bytes in 340 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 04:59:40.152603 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e280471e-6bc0-4915-886a-c219160397fd None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:40.175219 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:40.187806 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7642de0d-27d2-4717-8a76-35f38d238125 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:40.188679 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7642de0d-27d2-4717-8a76-35f38d238125 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:40.189353 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7642de0d-27d2-4717-8a76-35f38d238125 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:40.190177 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7642de0d-27d2-4717-8a76-35f38d238125 None admin] PATH_INFO: `/v3/users` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:40.191124 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7642de0d-27d2-4717-8a76-35f38d238125 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:40.191615 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7642de0d-27d2-4717-8a76-35f38d238125 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:40.193771 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7642de0d-27d2-4717-8a76-35f38d238125 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:40.203900 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:40.226204 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 281/552] 10.222.0.77 () {56 vars in 1135 bytes} [Tue Aug 30 04:59:39 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 267 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:40.226204 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-266c9f39-d492-48c7-80e5-d6673e72ca53 None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:40.226204 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-266c9f39-d492-48c7-80e5-d6673e72ca53 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:40.226204 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-266c9f39-d492-48c7-80e5-d6673e72ca53 None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:40.226204 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 282/553] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:59:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:59:40.229005 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-266c9f39-d492-48c7-80e5-d6673e72ca53 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:40.292161 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e280471e-6bc0-4915-886a-c219160397fd None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:40.338713 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a1e76fd-0bce-4467-b39d-cc85e51dff6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:40.339357 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a1e76fd-0bce-4467-b39d-cc85e51dff6a None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:40.339640 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a1e76fd-0bce-4467-b39d-cc85e51dff6a None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:40.339916 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a1e76fd-0bce-4467-b39d-cc85e51dff6a None admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:40.340732 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a1e76fd-0bce-4467-b39d-cc85e51dff6a None admin] RBAC: Adding query filter params (name=service) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:40.341106 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a1e76fd-0bce-4467-b39d-cc85e51dff6a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:40.342966 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a1e76fd-0bce-4467-b39d-cc85e51dff6a None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:40.358226 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 272/554] 10.222.0.77 () {56 vars in 1103 bytes} [Tue Aug 30 04:59:40 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:59:40.358226 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d423e6-d1ff-4e01-a2b0-612249fd2b32 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:40.358226 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d423e6-d1ff-4e01-a2b0-612249fd2b32 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:40.358226 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d423e6-d1ff-4e01-a2b0-612249fd2b32 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:40.369596 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-266c9f39-d492-48c7-80e5-d6673e72ca53 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:40.377215 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-e6d423e6-d1ff-4e01-a2b0-612249fd2b32 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:59:40.378418 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-e6d423e6-d1ff-4e01-a2b0-612249fd2b32 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:59:40.409302 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4d047cc-d73d-4889-a1e9-86d301cc015c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:40.409546 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4d047cc-d73d-4889-a1e9-86d301cc015c None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:40.409828 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4d047cc-d73d-4889-a1e9-86d301cc015c None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:40.410095 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4d047cc-d73d-4889-a1e9-86d301cc015c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:40.446629 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:40.454731 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 273/555] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:59:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:59:40.455619 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6d423e6-d1ff-4e01-a2b0-612249fd2b32 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:40.489719 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:40.494266 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4d047cc-d73d-4889-a1e9-86d301cc015c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:40.495777 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4d047cc-d73d-4889-a1e9-86d301cc015c None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:40.587058 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6d423e6-d1ff-4e01-a2b0-612249fd2b32 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:40.611982 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 283/556] 10.222.0.77 () {58 vars in 1282 bytes} [Tue Aug 30 04:59:40 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 379 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 04:59:40.611982 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b3033a-2a43-4a03-8aa3-9d2ee9e5145c None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:40.611982 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b3033a-2a43-4a03-8aa3-9d2ee9e5145c None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:40.611982 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b3033a-2a43-4a03-8aa3-9d2ee9e5145c None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:40.614002 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 284/557] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:59:40 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:59:40.614002 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f168ec-c057-42fd-853a-246918420a9d None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:40.614122 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f168ec-c057-42fd-853a-246918420a9d None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:40.616491 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f168ec-c057-42fd-853a-246918420a9d None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:40.616871 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 285/558] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:59:40 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:59:40.618407 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78f168ec-c057-42fd-853a-246918420a9d None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:40.628109 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef4946d2-18e1-4a2e-ba3d-be84bc79546d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:40.628863 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef4946d2-18e1-4a2e-ba3d-be84bc79546d None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:40.629565 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef4946d2-18e1-4a2e-ba3d-be84bc79546d None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:40.629908 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef4946d2-18e1-4a2e-ba3d-be84bc79546d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:40.640690 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef4946d2-18e1-4a2e-ba3d-be84bc79546d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:40.642176 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef4946d2-18e1-4a2e-ba3d-be84bc79546d None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:40.656904 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-ef4946d2-18e1-4a2e-ba3d-be84bc79546d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:59:40.656904 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:40.656904 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:40.656904 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:40.656904 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:40.656904 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:40.656904 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:40.656904 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:40.656904 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:40.656904 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:40.656904 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:40.656904 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:40.656904 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:59:40.656904 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:59:40.656904 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:59:40.656904 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:59:40.657853 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:40.657853 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:40.657853 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:40.657853 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:40.657853 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:40.657853 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:40.657853 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:40.657853 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:40.657853 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:40.657853 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:40.657853 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:40.657853 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:40.657853 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:40.657853 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:40.657853 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:40.657853 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:40.657853 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:59:40.658796 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:59:40.658796 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:59:40.658796 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 04:59:40.658796 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 274/559] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 04:59:40 2022] GET /identity/v3/domains/Default => generated 87 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:40.659455 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0164ad9d-5013-4ab9-a14a-9534e092fa45 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:40.659550 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0164ad9d-5013-4ab9-a14a-9534e092fa45 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:40.659780 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0164ad9d-5013-4ab9-a14a-9534e092fa45 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:40.678548 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-0164ad9d-5013-4ab9-a14a-9534e092fa45 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:59:40.680392 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-0164ad9d-5013-4ab9-a14a-9534e092fa45 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:59:40.695659 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:40.733708 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78f168ec-c057-42fd-853a-246918420a9d None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:40.750790 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 275/560] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:59:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:59:40.752097 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ef99ddd-ea35-4ec3-9aa1-8997ab5c4cda None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:40.752385 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ef99ddd-ea35-4ec3-9aa1-8997ab5c4cda None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:40.752655 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ef99ddd-ea35-4ec3-9aa1-8997ab5c4cda None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:40.764560 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:40.770649 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-2ef99ddd-ea35-4ec3-9aa1-8997ab5c4cda None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:59:40.772224 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-2ef99ddd-ea35-4ec3-9aa1-8997ab5c4cda None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:59:40.780063 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-860e8ab4-909d-4149-b10c-fdaff22ba8ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:40.781333 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-860e8ab4-909d-4149-b10c-fdaff22ba8ee None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:40.781795 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-860e8ab4-909d-4149-b10c-fdaff22ba8ee None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:40.782246 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-860e8ab4-909d-4149-b10c-fdaff22ba8ee None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:40.795869 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-860e8ab4-909d-4149-b10c-fdaff22ba8ee None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:40.798210 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-860e8ab4-909d-4149-b10c-fdaff22ba8ee None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:40.817040 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-860e8ab4-909d-4149-b10c-fdaff22ba8ee None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:59:40.817040 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:40.817040 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:40.817040 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:40.817040 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:40.817040 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:40.817040 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:40.817040 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:40.817040 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:40.817040 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:40.817040 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:40.817040 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:40.817040 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:59:40.817040 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:59:40.817040 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:59:40.817040 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:59:40.818203 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:40.818203 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:40.818203 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:40.818203 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:40.818203 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:40.818203 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:40.818203 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:40.818203 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:40.818203 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:40.818203 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:40.818203 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:40.818203 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:40.818203 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:40.818203 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:40.818203 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:40.818203 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:40.818203 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:59:40.819519 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:59:40.819519 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:59:40.819519 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 04:59:40.821943 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 286/561] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 04:59:40 2022] GET /identity/v3/domains/Default => generated 87 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:40.823151 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-860e8ab4-909d-4149-b10c-fdaff22ba8ee None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:40.840952 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 276/562] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:59:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:59:40.841711 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ef99ddd-ea35-4ec3-9aa1-8997ab5c4cda None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:40.952624 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-860e8ab4-909d-4149-b10c-fdaff22ba8ee None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:40.964211 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ef99ddd-ea35-4ec3-9aa1-8997ab5c4cda None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:40.973162 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:40.989247 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fc89380-7d64-49c3-95b6-52668e7e70c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:40.989576 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc89380-7d64-49c3-95b6-52668e7e70c6 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:40.989908 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc89380-7d64-49c3-95b6-52668e7e70c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:40.990183 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc89380-7d64-49c3-95b6-52668e7e70c6 None admin] PATH_INFO: `/v3/domains` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:40.990835 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fc89380-7d64-49c3-95b6-52668e7e70c6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:40.991147 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fc89380-7d64-49c3-95b6-52668e7e70c6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:40.992556 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fc89380-7d64-49c3-95b6-52668e7e70c6 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:41.005006 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 287/563] 10.222.0.77 () {56 vars in 1109 bytes} [Tue Aug 30 04:59:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:41.005184 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c25cc67a-280f-4a11-98f7-0d5900009c0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:41.005861 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fc89380-7d64-49c3-95b6-52668e7e70c6 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:41.006215 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c25cc67a-280f-4a11-98f7-0d5900009c0e None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:41.006493 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c25cc67a-280f-4a11-98f7-0d5900009c0e None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:41.006759 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c25cc67a-280f-4a11-98f7-0d5900009c0e None admin] PATH_INFO: `/v3/roles/service` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:41.016082 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:41.027024 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c25cc67a-280f-4a11-98f7-0d5900009c0e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:59:41.031854 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-c25cc67a-280f-4a11-98f7-0d5900009c0e None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 30 04:59:41.031854 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:41.031854 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:41.031854 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:41.031854 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:41.031854 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:41.031854 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:41.031854 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:41.031854 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:41.031854 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:41.031854 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:41.031854 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:41.031854 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:59:41.031854 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:59:41.031854 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:59:41.031854 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:59:41.032952 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:59:41.032952 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:59:41.032952 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:59:41.032952 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:59:41.032952 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:41.032952 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:41.032952 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:41.032952 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:41.032952 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:41.032952 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:41.032952 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:41.032952 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:41.032952 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:41.032952 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:41.032952 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:41.032952 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:41.033790 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:41.033790 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:41.033790 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:41.033790 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:41.033790 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:59:41.033790 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:59:41.033790 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:59:41.033790 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:59:41.033790 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:59:41.033790 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:59:41.033790 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 30 04:59:41.033790 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 04:59:41.033790 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 277/564] 10.222.0.77 () {56 vars in 1102 bytes} [Tue Aug 30 04:59:40 2022] GET /identity/v3/roles/service => generated 85 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:41.034464 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c25cc67a-280f-4a11-98f7-0d5900009c0e None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:41.114078 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fc89380-7d64-49c3-95b6-52668e7e70c6 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:41.140210 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c25cc67a-280f-4a11-98f7-0d5900009c0e None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:41.149127 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-388e0e6f-94e2-417b-b62c-605c3a1eb6fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:41.149719 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-388e0e6f-94e2-417b-b62c-605c3a1eb6fe None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:41.149998 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-388e0e6f-94e2-417b-b62c-605c3a1eb6fe None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:41.150263 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-388e0e6f-94e2-417b-b62c-605c3a1eb6fe None admin] PATH_INFO: `/v3/domains` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:41.150908 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-388e0e6f-94e2-417b-b62c-605c3a1eb6fe None admin] RBAC: Adding query filter params (name=Default) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:41.151203 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-388e0e6f-94e2-417b-b62c-605c3a1eb6fe None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:41.152561 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-388e0e6f-94e2-417b-b62c-605c3a1eb6fe None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:41.168186 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 288/565] 10.222.0.77 () {56 vars in 1109 bytes} [Tue Aug 30 04:59:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:41.168186 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-388e0e6f-94e2-417b-b62c-605c3a1eb6fe None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:41.172630 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13bda196-f7f1-4328-b9ed-369a77c09e49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:41.173245 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13bda196-f7f1-4328-b9ed-369a77c09e49 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:41.173524 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13bda196-f7f1-4328-b9ed-369a77c09e49 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:41.173790 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13bda196-f7f1-4328-b9ed-369a77c09e49 None admin] PATH_INFO: `/v3/users/glance` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:41.181628 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13bda196-f7f1-4328-b9ed-369a77c09e49 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:41.183094 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13bda196-f7f1-4328-b9ed-369a77c09e49 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:41.193925 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-13bda196-f7f1-4328-b9ed-369a77c09e49 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Aug 30 04:59:41.193925 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:41.193925 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:41.193925 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:41.193925 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:41.193925 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:41.193925 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:41.193925 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:41.193925 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:41.193925 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:41.193925 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:41.193925 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:41.193925 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 04:59:41.193925 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 04:59:41.193925 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 04:59:41.193925 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 04:59:41.194896 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:41.194896 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:41.194896 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 04:59:41.194896 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:59:41.194896 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 04:59:41.194896 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:59:41.194896 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:41.194896 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:41.194896 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:41.194896 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:41.194896 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:41.194896 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:41.194896 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:41.194896 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:41.194896 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:41.194896 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:41.194896 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:41.194896 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:41.196246 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:41.196246 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:41.196246 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 04:59:41.196246 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 04:59:41.196246 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 04:59:41.196246 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 04:59:41.196246 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 04:59:41.196246 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 04:59:41.196246 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Aug 30 04:59:41.196246 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 04:59:41.196246 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 278/566] 10.222.0.77 () {56 vars in 1099 bytes} [Tue Aug 30 04:59:41 2022] GET /identity/v3/users/glance => generated 84 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:41.196246 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13bda196-f7f1-4328-b9ed-369a77c09e49 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:41.214007 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:41.250090 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:41.272219 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-388e0e6f-94e2-417b-b62c-605c3a1eb6fe None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:41.294169 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13bda196-f7f1-4328-b9ed-369a77c09e49 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:41.305252 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-498f5900-68a7-46cc-8453-dcb60995c56b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:41.305862 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-498f5900-68a7-46cc-8453-dcb60995c56b None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:41.306138 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-498f5900-68a7-46cc-8453-dcb60995c56b None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:41.306407 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-498f5900-68a7-46cc-8453-dcb60995c56b None admin] PATH_INFO: `/v3/roles/service` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:41.316181 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-498f5900-68a7-46cc-8453-dcb60995c56b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:59:41.321369 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-498f5900-68a7-46cc-8453-dcb60995c56b None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 30 04:59:41.321369 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:41.321369 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:41.321369 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:41.321369 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:41.321369 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:41.321369 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:41.321369 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:41.321369 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:41.321369 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:41.321369 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:41.321369 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:41.321369 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:59:41.321369 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:59:41.321369 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:59:41.321369 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:59:41.326956 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:59:41.326956 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:59:41.326956 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:59:41.326956 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:59:41.326956 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:41.326956 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:41.326956 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:41.326956 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:41.326956 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:41.326956 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:41.326956 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:41.326956 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:41.326956 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:41.326956 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:41.326956 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:41.326956 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:41.328025 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:41.328025 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:41.328025 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:41.328025 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:41.328025 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:59:41.328025 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:59:41.328025 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:59:41.328025 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:59:41.328025 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:59:41.328025 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:59:41.328025 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 30 04:59:41.328025 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 04:59:41.328025 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 289/567] 10.222.0.77 () {56 vars in 1102 bytes} [Tue Aug 30 04:59:41 2022] GET /identity/v3/roles/service => generated 85 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:41.328025 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-498f5900-68a7-46cc-8453-dcb60995c56b None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:41.331895 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ce60332-4867-4752-bc56-c0dcc58f2756 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:41.332605 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ce60332-4867-4752-bc56-c0dcc58f2756 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:41.332884 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ce60332-4867-4752-bc56-c0dcc58f2756 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:41.333154 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ce60332-4867-4752-bc56-c0dcc58f2756 None admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:41.333805 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ce60332-4867-4752-bc56-c0dcc58f2756 None admin] RBAC: Adding query filter params (name=service) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:41.334103 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ce60332-4867-4752-bc56-c0dcc58f2756 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:41.335515 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ce60332-4867-4752-bc56-c0dcc58f2756 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:41.347669 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 279/568] 10.222.0.77 () {56 vars in 1103 bytes} [Tue Aug 30 04:59:41 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:41.347669 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ce60332-4867-4752-bc56-c0dcc58f2756 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:41.423065 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:41.426595 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-498f5900-68a7-46cc-8453-dcb60995c56b None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:41.441486 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ce60332-4867-4752-bc56-c0dcc58f2756 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:41.459881 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7a26896-0677-41e1-8e81-be68fc673e9c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:41.460531 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a26896-0677-41e1-8e81-be68fc673e9c None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:41.460810 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a26896-0677-41e1-8e81-be68fc673e9c None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:41.461078 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a26896-0677-41e1-8e81-be68fc673e9c None admin] PATH_INFO: `/v3/roles/service` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:41.461203 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:41.473188 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7a26896-0677-41e1-8e81-be68fc673e9c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:59:41.478858 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a02e47c-8062-43f3-b33c-0053ba608642 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:41.479221 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-e7a26896-0677-41e1-8e81-be68fc673e9c None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 30 04:59:41.479221 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:41.479221 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:41.479221 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:41.479221 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:41.479221 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:41.479221 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:41.479221 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:41.479221 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:41.479221 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:41.479221 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:41.479221 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:41.479221 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:59:41.479221 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:59:41.479221 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:59:41.479221 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:59:41.480206 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:59:41.480206 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:59:41.480206 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:59:41.480206 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:59:41.480206 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:41.480206 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:41.480206 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:41.480206 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:41.480206 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:41.480206 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:41.480206 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:41.480206 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:41.480206 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:41.480206 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:41.480206 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:41.480206 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:41.481703 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:41.481703 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:41.481703 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:41.481703 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:41.481703 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:59:41.481703 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:59:41.481703 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:59:41.481703 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:59:41.481703 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:59:41.481703 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:59:41.481703 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 30 04:59:41.481703 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 04:59:41.482347 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a02e47c-8062-43f3-b33c-0053ba608642 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:41.482347 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a02e47c-8062-43f3-b33c-0053ba608642 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:41.482347 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a02e47c-8062-43f3-b33c-0053ba608642 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:41.482521 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 290/569] 10.222.0.77 () {56 vars in 1102 bytes} [Tue Aug 30 04:59:41 2022] GET /identity/v3/roles/service => generated 85 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:41.482521 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7a26896-0677-41e1-8e81-be68fc673e9c None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:41.488146 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a02e47c-8062-43f3-b33c-0053ba608642 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:41.489587 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a02e47c-8062-43f3-b33c-0053ba608642 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:41.501225 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-0a02e47c-8062-43f3-b33c-0053ba608642 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:59:41.501225 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:41.501225 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:41.501225 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:41.501225 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:41.501225 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:41.501225 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:41.501225 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:41.501225 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:41.501225 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:41.501225 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:41.501225 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:41.501225 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 04:59:41.501225 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 04:59:41.501225 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 04:59:41.502166 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 04:59:41.502166 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:41.502166 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:41.502166 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:41.502166 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:41.502166 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:41.502166 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:41.502166 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:41.502166 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:41.502166 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:41.502166 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:41.502166 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:41.502166 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:41.502166 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:41.502166 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:41.502166 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:41.502166 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:41.503058 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 04:59:41.503058 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 04:59:41.503058 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 04:59:41.503058 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 04:59:41.503058 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 04:59:41.503058 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 04:59:41.503058 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:59:41.503058 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 04:59:41.503058 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 280/570] 10.222.0.77 () {56 vars in 1111 bytes} [Tue Aug 30 04:59:41 2022] GET /identity/v3/projects/service => generated 88 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:41.503536 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a02e47c-8062-43f3-b33c-0053ba608642 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:41.582115 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7a26896-0677-41e1-8e81-be68fc673e9c None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:41.598537 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a02e47c-8062-43f3-b33c-0053ba608642 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:41.615869 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20fa35c4-9163-457d-90fb-f9c6250370a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:41.616453 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20fa35c4-9163-457d-90fb-f9c6250370a3 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:41.616723 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20fa35c4-9163-457d-90fb-f9c6250370a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:41.616986 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20fa35c4-9163-457d-90fb-f9c6250370a3 None admin] PATH_INFO: `/v3/users` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:41.617691 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20fa35c4-9163-457d-90fb-f9c6250370a3 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:41.617991 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20fa35c4-9163-457d-90fb-f9c6250370a3 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:41.619448 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20fa35c4-9163-457d-90fb-f9c6250370a3 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:41.631380 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1dc2b59-8b8e-49bf-b8de-f90810661418 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:41.631965 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1dc2b59-8b8e-49bf-b8de-f90810661418 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:41.632248 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1dc2b59-8b8e-49bf-b8de-f90810661418 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:41.632553 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1dc2b59-8b8e-49bf-b8de-f90810661418 None admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:41.633195 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1dc2b59-8b8e-49bf-b8de-f90810661418 None admin] RBAC: Adding query filter params (name=service) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:41.633491 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1dc2b59-8b8e-49bf-b8de-f90810661418 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:41.634837 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1dc2b59-8b8e-49bf-b8de-f90810661418 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:41.635742 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 291/571] 10.222.0.77 () {56 vars in 1137 bytes} [Tue Aug 30 04:59:41 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:41.636513 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20fa35c4-9163-457d-90fb-f9c6250370a3 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:41.651861 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 281/572] 10.222.0.77 () {56 vars in 1103 bytes} [Tue Aug 30 04:59:41 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:41.651861 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1dc2b59-8b8e-49bf-b8de-f90810661418 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:41.655487 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:41.688374 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:41.736428 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20fa35c4-9163-457d-90fb-f9c6250370a3 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:41.745465 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1dc2b59-8b8e-49bf-b8de-f90810661418 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:41.768879 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baa71b1e-f604-4344-baf4-44578b2b6ed0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:41.769496 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa71b1e-f604-4344-baf4-44578b2b6ed0 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:41.769771 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa71b1e-f604-4344-baf4-44578b2b6ed0 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:41.770041 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa71b1e-f604-4344-baf4-44578b2b6ed0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:41.777727 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-902ca54f-2b32-4a99-b914-fd3aa60f7308 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:41.778303 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-902ca54f-2b32-4a99-b914-fd3aa60f7308 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:41.778580 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-902ca54f-2b32-4a99-b914-fd3aa60f7308 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:41.778871 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-902ca54f-2b32-4a99-b914-fd3aa60f7308 None admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:41.779515 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-902ca54f-2b32-4a99-b914-fd3aa60f7308 None admin] RBAC: Adding query filter params (name=service) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:41.779809 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-902ca54f-2b32-4a99-b914-fd3aa60f7308 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:41.780535 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baa71b1e-f604-4344-baf4-44578b2b6ed0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:41.781164 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-902ca54f-2b32-4a99-b914-fd3aa60f7308 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:41.781965 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baa71b1e-f604-4344-baf4-44578b2b6ed0 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:41.789309 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 282/573] 10.222.0.77 () {56 vars in 1103 bytes} [Tue Aug 30 04:59:41 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:41.790079 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-902ca54f-2b32-4a99-b914-fd3aa60f7308 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:41.798473 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-baa71b1e-f604-4344-baf4-44578b2b6ed0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:59:41.798473 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:41.798473 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:41.798473 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:41.798473 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:41.798473 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:41.798473 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:41.798473 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:41.798473 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:41.798473 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:41.798473 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:41.798473 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:41.798473 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:59:41.798473 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:59:41.798473 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:59:41.798473 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:59:41.799627 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:41.799627 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:41.799627 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:41.799627 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:41.799627 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:41.799627 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:41.799627 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:41.799627 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:41.799627 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:41.799627 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:41.799627 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:41.799627 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:41.799627 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:41.799627 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:41.799627 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:41.799627 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:41.799627 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:59:41.800534 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:59:41.800534 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:59:41.800534 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 04:59:41.800534 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 292/574] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 04:59:41 2022] GET /identity/v3/domains/Default => generated 87 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:41.800768 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baa71b1e-f604-4344-baf4-44578b2b6ed0 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:41.868746 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:41.890890 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-902ca54f-2b32-4a99-b914-fd3aa60f7308 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:41.892725 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:41.898844 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baa71b1e-f604-4344-baf4-44578b2b6ed0 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:41.925659 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc75ba2b-6c1e-4887-b962-d4bda88af31f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:41.926247 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc75ba2b-6c1e-4887-b962-d4bda88af31f None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:41.926521 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc75ba2b-6c1e-4887-b962-d4bda88af31f None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:41.926787 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc75ba2b-6c1e-4887-b962-d4bda88af31f None admin] PATH_INFO: `/v3/projects` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:41.927484 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc75ba2b-6c1e-4887-b962-d4bda88af31f None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:41.927783 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc75ba2b-6c1e-4887-b962-d4bda88af31f None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:41.929263 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc75ba2b-6c1e-4887-b962-d4bda88af31f None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:41.931654 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e0439b7-ee64-486b-abde-973a45c3741c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:41.932262 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e0439b7-ee64-486b-abde-973a45c3741c None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:41.932540 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e0439b7-ee64-486b-abde-973a45c3741c None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:41.932804 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e0439b7-ee64-486b-abde-973a45c3741c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:41.940964 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 283/575] 10.222.0.77 () {56 vars in 1148 bytes} [Tue Aug 30 04:59:41 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:41.941671 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc75ba2b-6c1e-4887-b962-d4bda88af31f None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:41.943104 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e0439b7-ee64-486b-abde-973a45c3741c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:41.944542 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e0439b7-ee64-486b-abde-973a45c3741c None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:41.959041 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-8e0439b7-ee64-486b-abde-973a45c3741c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:59:41.959041 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:41.959041 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:41.959041 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:41.959041 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:41.959041 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:41.959041 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:41.959041 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:41.959041 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:41.959041 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:41.959041 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:41.959041 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:41.959041 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:59:41.959041 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:59:41.959041 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:59:41.959041 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:59:41.965814 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:41.965814 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:41.965814 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:41.965814 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:41.965814 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:41.965814 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:41.965814 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:41.965814 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:41.965814 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:41.965814 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:41.965814 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:41.965814 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:41.965814 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:41.965814 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:41.965814 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:41.965814 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:41.965814 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:59:41.966953 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:59:41.966953 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:59:41.966953 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 04:59:41.966953 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 293/576] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 04:59:41 2022] GET /identity/v3/domains/Default => generated 87 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:41.966953 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e0439b7-ee64-486b-abde-973a45c3741c None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:42.041428 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc75ba2b-6c1e-4887-b962-d4bda88af31f None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:42.061975 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e0439b7-ee64-486b-abde-973a45c3741c None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:42.074111 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2e3645e-a2d6-4e2a-a96f-537bd2b51816 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:42.074716 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2e3645e-a2d6-4e2a-a96f-537bd2b51816 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:42.075023 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2e3645e-a2d6-4e2a-a96f-537bd2b51816 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:42.075294 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2e3645e-a2d6-4e2a-a96f-537bd2b51816 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:42.084426 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:42.085407 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2e3645e-a2d6-4e2a-a96f-537bd2b51816 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:42.087522 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2e3645e-a2d6-4e2a-a96f-537bd2b51816 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:42.097299 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da443982-0ce6-408e-a17e-78d7b43e24a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:42.097907 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da443982-0ce6-408e-a17e-78d7b43e24a2 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:42.098207 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da443982-0ce6-408e-a17e-78d7b43e24a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:42.098474 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da443982-0ce6-408e-a17e-78d7b43e24a2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:42.102160 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-d2e3645e-a2d6-4e2a-a96f-537bd2b51816 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:59:42.102160 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:42.102160 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:42.102160 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:42.102160 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:42.102160 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:42.102160 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:42.102160 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:42.102160 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:42.102160 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:42.102160 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:42.102160 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:42.102160 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:59:42.102160 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:59:42.102160 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:59:42.102160 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:59:42.103533 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:42.103533 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:42.103533 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:42.103533 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:42.103533 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:42.103533 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:42.103533 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:42.103533 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:42.103533 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:42.103533 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:42.103533 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:42.103533 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:42.103533 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:42.103533 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:42.103533 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:42.103533 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:42.103533 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:59:42.104621 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:59:42.104621 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:59:42.104621 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 04:59:42.108204 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 284/577] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 04:59:41 2022] GET /identity/v3/domains/Default => generated 87 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:42.108809 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2e3645e-a2d6-4e2a-a96f-537bd2b51816 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:42.109236 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da443982-0ce6-408e-a17e-78d7b43e24a2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:42.110664 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da443982-0ce6-408e-a17e-78d7b43e24a2 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:42.124008 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:42.124768 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-da443982-0ce6-408e-a17e-78d7b43e24a2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:59:42.124768 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:42.124768 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:42.124768 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:42.124768 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:42.124768 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:42.124768 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:42.124768 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:42.124768 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:42.124768 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:42.124768 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:42.124768 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:42.124768 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:59:42.124768 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:59:42.124768 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:59:42.124768 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:59:42.126042 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:42.126042 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:42.126042 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:42.126042 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:42.126042 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:42.126042 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:42.126042 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:42.126042 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:42.126042 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:42.126042 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:42.126042 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:42.126042 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:42.126042 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:42.126042 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:42.126042 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:42.126042 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:42.126042 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:59:42.127266 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:59:42.127266 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:59:42.127266 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 04:59:42.127266 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 294/578] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 04:59:41 2022] GET /identity/v3/domains/Default => generated 87 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:42.127266 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da443982-0ce6-408e-a17e-78d7b43e24a2 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:42.210266 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2e3645e-a2d6-4e2a-a96f-537bd2b51816 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:42.224175 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da443982-0ce6-408e-a17e-78d7b43e24a2 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:42.242737 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93276ce9-5470-4e29-9a77-ea091c787c01 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:42.243369 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93276ce9-5470-4e29-9a77-ea091c787c01 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:42.243649 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93276ce9-5470-4e29-9a77-ea091c787c01 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:42.243918 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93276ce9-5470-4e29-9a77-ea091c787c01 None admin] PATH_INFO: `/v3/domains` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:42.244549 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93276ce9-5470-4e29-9a77-ea091c787c01 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:42.244846 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93276ce9-5470-4e29-9a77-ea091c787c01 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:42.246211 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93276ce9-5470-4e29-9a77-ea091c787c01 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:42.256523 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86c2c029-611c-4839-bf35-7cd589aa2c26 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:42.257105 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86c2c029-611c-4839-bf35-7cd589aa2c26 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:42.257865 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86c2c029-611c-4839-bf35-7cd589aa2c26 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:42.258113 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 285/579] 10.222.0.77 () {56 vars in 1109 bytes} [Tue Aug 30 04:59:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:42.258227 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86c2c029-611c-4839-bf35-7cd589aa2c26 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:42.258847 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86c2c029-611c-4839-bf35-7cd589aa2c26 None admin] RBAC: Adding query filter params (role.id=821647e8b40542499667e29e1b301cb2, scope.project.id=e9919141e3b947ab8d91b55bba4a7de6, user.id=e0a7f7afc3d44ec788c05d005e52a1b8) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:42.258951 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93276ce9-5470-4e29-9a77-ea091c787c01 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:42.259163 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86c2c029-611c-4839-bf35-7cd589aa2c26 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:42.260585 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86c2c029-611c-4839-bf35-7cd589aa2c26 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:42.275082 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 295/580] 10.222.0.77 () {56 vars in 1374 bytes} [Tue Aug 30 04:59:42 2022] GET /identity/v3/role_assignments?user.id=e0a7f7afc3d44ec788c05d005e52a1b8&scope.project.id=e9919141e3b947ab8d91b55bba4a7de6&role.id=821647e8b40542499667e29e1b301cb2 => generated 259 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:42.275082 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86c2c029-611c-4839-bf35-7cd589aa2c26 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:42.318941 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:42.344771 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:42.376510 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93276ce9-5470-4e29-9a77-ea091c787c01 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:42.387535 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86c2c029-611c-4839-bf35-7cd589aa2c26 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:42.428204 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eff603d-74d8-4dc1-8599-c505cb31561b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:42.428595 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-124e31dc-a930-41d7-8f77-6a6a6f78c826 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:42.429165 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-124e31dc-a930-41d7-8f77-6a6a6f78c826 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:42.429958 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-124e31dc-a930-41d7-8f77-6a6a6f78c826 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:42.430226 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-124e31dc-a930-41d7-8f77-6a6a6f78c826 None admin] PATH_INFO: `/v3/domains` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:42.430473 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eff603d-74d8-4dc1-8599-c505cb31561b None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:42.430892 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-124e31dc-a930-41d7-8f77-6a6a6f78c826 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:42.431138 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eff603d-74d8-4dc1-8599-c505cb31561b None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:42.431239 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-124e31dc-a930-41d7-8f77-6a6a6f78c826 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:42.431668 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eff603d-74d8-4dc1-8599-c505cb31561b None admin] PATH_INFO: `/v3/domains` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:42.432488 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eff603d-74d8-4dc1-8599-c505cb31561b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:42.432608 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-124e31dc-a930-41d7-8f77-6a6a6f78c826 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:42.433072 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eff603d-74d8-4dc1-8599-c505cb31561b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:42.434734 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eff603d-74d8-4dc1-8599-c505cb31561b None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:42.444586 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 296/581] 10.222.0.77 () {56 vars in 1109 bytes} [Tue Aug 30 04:59:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:42.445960 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-124e31dc-a930-41d7-8f77-6a6a6f78c826 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:42.456698 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 286/582] 10.222.0.77 () {56 vars in 1109 bytes} [Tue Aug 30 04:59:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:42.457863 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eff603d-74d8-4dc1-8599-c505cb31561b None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:42.548070 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:42.552969 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-124e31dc-a930-41d7-8f77-6a6a6f78c826 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:42.567215 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eff603d-74d8-4dc1-8599-c505cb31561b None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:42.597103 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42ce2698-737a-410d-9dec-f63c51cc7cff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:42.598132 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ce2698-737a-410d-9dec-f63c51cc7cff None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:42.598651 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ce2698-737a-410d-9dec-f63c51cc7cff None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:42.599183 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ce2698-737a-410d-9dec-f63c51cc7cff None admin] PATH_INFO: `/v3/domains` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:42.599822 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42ce2698-737a-410d-9dec-f63c51cc7cff None admin] RBAC: Adding query filter params (name=Default) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:42.600124 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42ce2698-737a-410d-9dec-f63c51cc7cff None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:42.601307 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf8d12e8-1cb0-4855-ad73-acdfc774254a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:42.601518 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42ce2698-737a-410d-9dec-f63c51cc7cff None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:42.602316 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf8d12e8-1cb0-4855-ad73-acdfc774254a None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:42.602918 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf8d12e8-1cb0-4855-ad73-acdfc774254a None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:42.603625 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf8d12e8-1cb0-4855-ad73-acdfc774254a None admin] PATH_INFO: `/v3/users/nova` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:42.605829 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:42.612270 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf8d12e8-1cb0-4855-ad73-acdfc774254a None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:42.618248 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 297/583] 10.222.0.77 () {56 vars in 1109 bytes} [Tue Aug 30 04:59:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:42.618248 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42ce2698-737a-410d-9dec-f63c51cc7cff None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:42.618497 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf8d12e8-1cb0-4855-ad73-acdfc774254a None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:42.626952 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-bf8d12e8-1cb0-4855-ad73-acdfc774254a None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 30 04:59:42.626952 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:42.626952 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:42.626952 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:42.626952 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:42.626952 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:42.626952 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:42.626952 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:42.626952 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:42.626952 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:42.626952 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:42.626952 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:42.626952 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 04:59:42.626952 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 04:59:42.626952 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 04:59:42.626952 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 04:59:42.628074 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:42.628074 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:42.628074 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 04:59:42.628074 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:59:42.628074 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 04:59:42.628074 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:59:42.628074 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:42.628074 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:42.628074 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:42.628074 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:42.628074 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:42.628074 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:42.628074 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:42.628074 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:42.628074 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:42.628074 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:42.628074 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:42.628074 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:42.629213 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:42.629213 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:42.629213 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 04:59:42.629213 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 04:59:42.629213 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 04:59:42.629213 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 04:59:42.629213 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 04:59:42.629213 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 04:59:42.629213 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 30 04:59:42.629213 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 04:59:42.629213 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 287/584] 10.222.0.77 () {56 vars in 1093 bytes} [Tue Aug 30 04:59:42 2022] GET /identity/v3/users/nova => generated 82 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:42.631425 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf8d12e8-1cb0-4855-ad73-acdfc774254a None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:42.728480 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42ce2698-737a-410d-9dec-f63c51cc7cff None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:42.744027 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf8d12e8-1cb0-4855-ad73-acdfc774254a None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:42.763895 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c75dbe91-6587-46c6-b492-9a7fda32eb1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:42.764967 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c75dbe91-6587-46c6-b492-9a7fda32eb1b None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:42.764967 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c75dbe91-6587-46c6-b492-9a7fda32eb1b None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:42.765267 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c75dbe91-6587-46c6-b492-9a7fda32eb1b None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:42.773115 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c75dbe91-6587-46c6-b492-9a7fda32eb1b None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:42.774648 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c75dbe91-6587-46c6-b492-9a7fda32eb1b None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:42.779383 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b01dbf76-0c52-4f14-b21e-dc9d9c6389f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:42.780181 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b01dbf76-0c52-4f14-b21e-dc9d9c6389f5 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:42.780607 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b01dbf76-0c52-4f14-b21e-dc9d9c6389f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:42.781027 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b01dbf76-0c52-4f14-b21e-dc9d9c6389f5 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:42.785774 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-c75dbe91-6587-46c6-b492-9a7fda32eb1b None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Aug 30 04:59:42.785774 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:42.785774 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:42.785774 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:42.785774 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:42.785774 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:42.785774 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:42.785774 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:42.785774 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:42.785774 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:42.785774 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:42.785774 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:42.785774 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 04:59:42.785774 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 04:59:42.785774 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 04:59:42.785774 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 04:59:42.786929 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:42.786929 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:42.786929 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 04:59:42.786929 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:59:42.786929 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 04:59:42.786929 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:59:42.786929 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:42.786929 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:42.786929 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:42.786929 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:42.786929 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:42.786929 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:42.786929 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:42.786929 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:42.786929 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:42.786929 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:42.786929 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:42.786929 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:42.788026 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:42.788026 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:42.788026 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 04:59:42.788026 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 04:59:42.788026 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 04:59:42.788026 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 04:59:42.788026 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 04:59:42.788026 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 04:59:42.788026 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Aug 30 04:59:42.788026 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 04:59:42.788026 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 298/585] 10.222.0.77 () {56 vars in 1099 bytes} [Tue Aug 30 04:59:42 2022] GET /identity/v3/users/cinder => generated 84 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:42.788026 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c75dbe91-6587-46c6-b492-9a7fda32eb1b None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:42.793174 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b01dbf76-0c52-4f14-b21e-dc9d9c6389f5 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:42.794775 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b01dbf76-0c52-4f14-b21e-dc9d9c6389f5 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:42.807161 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-b01dbf76-0c52-4f14-b21e-dc9d9c6389f5 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:59:42.807161 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:42.807161 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:42.807161 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:42.807161 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:42.807161 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:42.807161 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:42.807161 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:42.807161 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:42.807161 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:42.807161 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:42.807161 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:42.807161 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 04:59:42.807161 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 04:59:42.807161 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 04:59:42.808234 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 04:59:42.808234 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:42.808234 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:42.808234 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:42.808234 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:42.808234 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:42.808234 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:42.808234 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:42.808234 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:42.808234 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:42.808234 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:42.808234 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:42.808234 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:42.808234 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:42.808234 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:42.808234 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:42.808234 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:42.809296 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 04:59:42.809296 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 04:59:42.809296 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 04:59:42.809296 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 04:59:42.809296 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 04:59:42.809296 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 04:59:42.809296 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:59:42.809296 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 04:59:42.810966 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 288/586] 10.222.0.77 () {56 vars in 1111 bytes} [Tue Aug 30 04:59:42 2022] GET /identity/v3/projects/service => generated 88 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:42.811926 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b01dbf76-0c52-4f14-b21e-dc9d9c6389f5 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:42.817123 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:42.871495 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:42.905339 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c75dbe91-6587-46c6-b492-9a7fda32eb1b None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:42.925488 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b01dbf76-0c52-4f14-b21e-dc9d9c6389f5 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:42.939324 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83022c23-a350-40c5-b720-876813dd9316 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:42.939939 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83022c23-a350-40c5-b720-876813dd9316 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:42.940207 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83022c23-a350-40c5-b720-876813dd9316 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:42.940474 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83022c23-a350-40c5-b720-876813dd9316 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:42.947729 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83022c23-a350-40c5-b720-876813dd9316 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:42.949155 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83022c23-a350-40c5-b720-876813dd9316 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:42.959900 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-83022c23-a350-40c5-b720-876813dd9316 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Aug 30 04:59:42.959900 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:42.959900 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:42.959900 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:42.959900 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:42.959900 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:42.959900 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:42.959900 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:42.959900 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:42.959900 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:42.959900 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:42.959900 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:42.959900 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 04:59:42.959900 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 04:59:42.959900 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 04:59:42.959900 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 04:59:42.966139 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:42.966139 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:42.966139 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 04:59:42.966139 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:59:42.966139 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 04:59:42.966139 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:59:42.966139 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:42.966139 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:42.966139 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:42.966139 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:42.966139 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:42.966139 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:42.966139 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:42.966139 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:42.966139 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:42.966139 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:42.966139 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:42.966139 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:42.967174 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:42.967174 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:42.967174 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 04:59:42.967174 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 04:59:42.967174 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 04:59:42.967174 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 04:59:42.967174 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 04:59:42.967174 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 04:59:42.967174 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Aug 30 04:59:42.967174 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 04:59:42.967707 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-754a7cb1-10a8-49b9-b39b-b8f5473791d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:42.967707 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-754a7cb1-10a8-49b9-b39b-b8f5473791d6 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:42.967863 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 299/587] 10.222.0.77 () {56 vars in 1102 bytes} [Tue Aug 30 04:59:42 2022] GET /identity/v3/users/neutron => generated 85 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:42.967931 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-754a7cb1-10a8-49b9-b39b-b8f5473791d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:42.967931 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-754a7cb1-10a8-49b9-b39b-b8f5473791d6 None admin] PATH_INFO: `/v3/users` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:42.968050 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83022c23-a350-40c5-b720-876813dd9316 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:42.968125 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-754a7cb1-10a8-49b9-b39b-b8f5473791d6 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:42.968125 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-754a7cb1-10a8-49b9-b39b-b8f5473791d6 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:42.968125 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-754a7cb1-10a8-49b9-b39b-b8f5473791d6 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:42.988512 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 289/588] 10.222.0.77 () {56 vars in 1133 bytes} [Tue Aug 30 04:59:42 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:42.990113 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-754a7cb1-10a8-49b9-b39b-b8f5473791d6 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:43.049270 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:43.071453 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83022c23-a350-40c5-b720-876813dd9316 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:43.097236 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-754a7cb1-10a8-49b9-b39b-b8f5473791d6 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:43.106281 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6891cbeb-877f-46af-97af-e8f63d7e56fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:43.106912 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6891cbeb-877f-46af-97af-e8f63d7e56fc None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:43.107189 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6891cbeb-877f-46af-97af-e8f63d7e56fc None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:43.107461 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6891cbeb-877f-46af-97af-e8f63d7e56fc None admin] PATH_INFO: `/v3/users` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:43.108173 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6891cbeb-877f-46af-97af-e8f63d7e56fc None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:43.108472 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6891cbeb-877f-46af-97af-e8f63d7e56fc None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:43.109885 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6891cbeb-877f-46af-97af-e8f63d7e56fc None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:43.111973 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:43.131986 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 300/589] 10.222.0.77 () {56 vars in 1137 bytes} [Tue Aug 30 04:59:42 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:43.132987 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6891cbeb-877f-46af-97af-e8f63d7e56fc None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:43.136246 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c47ddac7-5964-40e6-9679-f866637055af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:43.137134 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c47ddac7-5964-40e6-9679-f866637055af None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:43.137134 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c47ddac7-5964-40e6-9679-f866637055af None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:43.137390 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c47ddac7-5964-40e6-9679-f866637055af None admin] PATH_INFO: `/v3/projects` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:43.138043 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c47ddac7-5964-40e6-9679-f866637055af None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:43.138342 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c47ddac7-5964-40e6-9679-f866637055af None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:43.139767 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c47ddac7-5964-40e6-9679-f866637055af None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:43.152209 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 290/590] 10.222.0.77 () {56 vars in 1148 bytes} [Tue Aug 30 04:59:42 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:43.152813 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c47ddac7-5964-40e6-9679-f866637055af None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:43.251517 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6891cbeb-877f-46af-97af-e8f63d7e56fc None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:43.257865 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c47ddac7-5964-40e6-9679-f866637055af None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:43.288139 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-402c1774-ee7c-4451-a266-7f41ca6c0fa2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:43.289272 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-402c1774-ee7c-4451-a266-7f41ca6c0fa2 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:43.289547 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-402c1774-ee7c-4451-a266-7f41ca6c0fa2 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:43.289812 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-402c1774-ee7c-4451-a266-7f41ca6c0fa2 None admin] PATH_INFO: `/v3/users` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:43.290518 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-402c1774-ee7c-4451-a266-7f41ca6c0fa2 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:43.290859 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-402c1774-ee7c-4451-a266-7f41ca6c0fa2 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:43.291988 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aa20d2e-c9ac-45aa-ab77-191cfa389780 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:43.292379 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-402c1774-ee7c-4451-a266-7f41ca6c0fa2 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:43.292740 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa20d2e-c9ac-45aa-ab77-191cfa389780 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:43.293086 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa20d2e-c9ac-45aa-ab77-191cfa389780 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:43.293423 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa20d2e-c9ac-45aa-ab77-191cfa389780 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:43.295172 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:43.305992 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aa20d2e-c9ac-45aa-ab77-191cfa389780 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:43.307564 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aa20d2e-c9ac-45aa-ab77-191cfa389780 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:43.316595 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 301/591] 10.222.0.77 () {56 vars in 1139 bytes} [Tue Aug 30 04:59:43 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:43.316595 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-402c1774-ee7c-4451-a266-7f41ca6c0fa2 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:43.326257 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-9aa20d2e-c9ac-45aa-ab77-191cfa389780 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:59:43.326257 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:43.326257 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:43.326257 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:43.326257 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:43.326257 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:43.326257 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:43.326257 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:43.326257 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:43.326257 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:43.326257 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:43.326257 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:43.326257 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:59:43.326257 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:59:43.326257 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:59:43.326257 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:59:43.327626 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:43.327626 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:43.327626 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:43.327626 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:43.327626 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:43.327626 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:43.327626 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:43.327626 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:43.327626 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:43.327626 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:43.327626 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:43.327626 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:43.327626 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:43.327626 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:43.327626 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:43.327626 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:43.327626 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:59:43.328707 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:59:43.328707 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:59:43.328707 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 04:59:43.331131 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 291/592] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 04:59:43 2022] GET /identity/v3/domains/Default => generated 87 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:43.331835 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aa20d2e-c9ac-45aa-ab77-191cfa389780 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:43.360161 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:43.423939 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-402c1774-ee7c-4451-a266-7f41ca6c0fa2 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:43.439060 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aa20d2e-c9ac-45aa-ab77-191cfa389780 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:43.467132 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0654e04d-0511-4262-8b86-b2dcad72b1f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:43.467871 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0654e04d-0511-4262-8b86-b2dcad72b1f9 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:43.468245 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0654e04d-0511-4262-8b86-b2dcad72b1f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:43.468649 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0654e04d-0511-4262-8b86-b2dcad72b1f9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:43.477677 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0220b0da-68ee-46ed-bb64-664b036fc9e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:43.478737 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0220b0da-68ee-46ed-bb64-664b036fc9e1 None admin] REQUEST_METHOD: `PUT` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:43.479053 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0220b0da-68ee-46ed-bb64-664b036fc9e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:43.479328 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0220b0da-68ee-46ed-bb64-664b036fc9e1 None admin] PATH_INFO: `/v3/projects/e9919141e3b947ab8d91b55bba4a7de6/users/e0a002ac0bf240a891ac7304178d3f0e/roles/821647e8b40542499667e29e1b301cb2` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:43.480207 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0654e04d-0511-4262-8b86-b2dcad72b1f9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:43.481702 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0654e04d-0511-4262-8b86-b2dcad72b1f9 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:43.494807 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0220b0da-68ee-46ed-bb64-664b036fc9e1 None admin] RBAC: Authorizing `identity:create_grant(project_id=e9919141e3b947ab8d91b55bba4a7de6, user_id=e0a002ac0bf240a891ac7304178d3f0e, role_id=821647e8b40542499667e29e1b301cb2)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:43.496279 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0220b0da-68ee-46ed-bb64-664b036fc9e1 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:43.496604 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-0654e04d-0511-4262-8b86-b2dcad72b1f9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:59:43.496604 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:43.496604 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:43.496604 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:43.496604 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:43.496604 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:43.496604 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:43.496604 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:43.496604 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:43.496604 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:43.496604 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:43.496604 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:43.496604 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:59:43.496604 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:59:43.496604 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:59:43.496604 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:59:43.497486 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:43.497486 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:43.497486 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:43.497486 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:43.497486 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:43.497486 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:43.497486 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:43.497486 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:43.497486 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:43.497486 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:43.497486 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:43.497486 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:43.497486 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:43.497486 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:43.497486 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:43.497486 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:43.497486 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:59:43.503337 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:59:43.503337 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:59:43.503337 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 04:59:43.503337 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 302/593] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 04:59:43 2022] GET /identity/v3/domains/Default => generated 87 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:43.503337 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0654e04d-0511-4262-8b86-b2dcad72b1f9 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:43.527073 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 292/594] 10.222.0.77 () {58 vars in 1439 bytes} [Tue Aug 30 04:59:43 2022] PUT /identity/v3/projects/e9919141e3b947ab8d91b55bba4a7de6/users/e0a002ac0bf240a891ac7304178d3f0e/roles/821647e8b40542499667e29e1b301cb2 => generated 0 bytes in 185 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:59:43.527073 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0220b0da-68ee-46ed-bb64-664b036fc9e1 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:43.566979 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:43.614199 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:43.646472 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0220b0da-68ee-46ed-bb64-664b036fc9e1 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:43.651946 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0654e04d-0511-4262-8b86-b2dcad72b1f9 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:43.685358 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37d06f3b-5528-4f1b-89be-9185b0cd268d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:43.687248 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37d06f3b-5528-4f1b-89be-9185b0cd268d None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:43.687248 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37d06f3b-5528-4f1b-89be-9185b0cd268d None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:43.687420 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37d06f3b-5528-4f1b-89be-9185b0cd268d None admin] PATH_INFO: `/v3/domains` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:43.688212 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37d06f3b-5528-4f1b-89be-9185b0cd268d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:43.688584 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37d06f3b-5528-4f1b-89be-9185b0cd268d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:43.690039 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37d06f3b-5528-4f1b-89be-9185b0cd268d None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:43.702269 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-520f5c5c-a57c-4daf-adcf-940993453de3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:43.702979 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-520f5c5c-a57c-4daf-adcf-940993453de3 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:43.703219 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-520f5c5c-a57c-4daf-adcf-940993453de3 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:43.703489 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-520f5c5c-a57c-4daf-adcf-940993453de3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:43.706150 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 293/595] 10.222.0.77 () {56 vars in 1109 bytes} [Tue Aug 30 04:59:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:43.707294 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37d06f3b-5528-4f1b-89be-9185b0cd268d None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:43.720573 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-520f5c5c-a57c-4daf-adcf-940993453de3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:43.722003 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-520f5c5c-a57c-4daf-adcf-940993453de3 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:43.739685 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-520f5c5c-a57c-4daf-adcf-940993453de3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:59:43.739685 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:43.739685 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:43.739685 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:43.739685 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:43.739685 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:43.739685 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:43.739685 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:43.739685 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:43.739685 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:43.739685 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:43.739685 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:43.739685 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:59:43.739685 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:59:43.739685 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:59:43.739685 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:59:43.741001 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:43.741001 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:43.741001 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:43.741001 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:43.741001 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:43.741001 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:43.741001 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:43.741001 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:43.741001 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:43.741001 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:43.741001 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:43.741001 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:43.741001 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:43.741001 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:43.741001 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:43.741001 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:43.741001 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:59:43.741976 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:59:43.741976 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:59:43.741976 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 04:59:43.741976 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 303/596] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 04:59:43 2022] GET /identity/v3/domains/Default => generated 87 bytes in 242 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:43.741976 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-520f5c5c-a57c-4daf-adcf-940993453de3 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:43.838859 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37d06f3b-5528-4f1b-89be-9185b0cd268d None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:43.857186 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:43.861382 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-520f5c5c-a57c-4daf-adcf-940993453de3 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:43.889179 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e55ca5d0-535e-427b-8f85-262020b5957b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:43.890350 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e55ca5d0-535e-427b-8f85-262020b5957b None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:43.890946 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e55ca5d0-535e-427b-8f85-262020b5957b None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:43.891750 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e55ca5d0-535e-427b-8f85-262020b5957b None admin] PATH_INFO: `/v3/domains` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:43.892853 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e55ca5d0-535e-427b-8f85-262020b5957b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:43.893573 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e55ca5d0-535e-427b-8f85-262020b5957b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:43.895939 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e55ca5d0-535e-427b-8f85-262020b5957b None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:43.897870 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:43.900883 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b62b641-38ad-4744-ab31-befa4371c97c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:43.901601 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b62b641-38ad-4744-ab31-befa4371c97c None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:43.901945 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b62b641-38ad-4744-ab31-befa4371c97c None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:43.902275 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b62b641-38ad-4744-ab31-befa4371c97c None admin] PATH_INFO: `/v3/projects/service` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:43.910569 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b62b641-38ad-4744-ab31-befa4371c97c None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:43.912051 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b62b641-38ad-4744-ab31-befa4371c97c None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:43.913433 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 294/597] 10.222.0.77 () {56 vars in 1109 bytes} [Tue Aug 30 04:59:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:43.915245 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e55ca5d0-535e-427b-8f85-262020b5957b None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:43.925957 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-8b62b641-38ad-4744-ab31-befa4371c97c None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:59:43.925957 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:43.925957 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:43.925957 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:43.925957 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:43.925957 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:43.925957 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:43.925957 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:43.925957 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:43.925957 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:43.925957 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:43.925957 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:43.925957 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 04:59:43.925957 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 04:59:43.925957 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 04:59:43.926936 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 04:59:43.926936 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:43.926936 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:43.926936 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:43.926936 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:43.926936 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:43.926936 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:43.926936 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:43.926936 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:43.926936 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:43.926936 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:43.926936 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:43.926936 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:43.926936 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:43.926936 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:43.926936 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:43.926936 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:43.928095 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 04:59:43.928095 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 04:59:43.928095 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 04:59:43.928095 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 04:59:43.928095 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 04:59:43.928095 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 04:59:43.928095 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:59:43.928095 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 04:59:43.928095 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 304/598] 10.222.0.77 () {56 vars in 1111 bytes} [Tue Aug 30 04:59:43 2022] GET /identity/v3/projects/service => generated 88 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:43.929705 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b62b641-38ad-4744-ab31-befa4371c97c None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:44.042032 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b62b641-38ad-4744-ab31-befa4371c97c None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:44.057564 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e55ca5d0-535e-427b-8f85-262020b5957b None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:44.085328 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4c3fd9f-49cd-4cb9-8d7a-9e09840b997e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:44.086520 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c3fd9f-49cd-4cb9-8d7a-9e09840b997e None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:44.087153 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c3fd9f-49cd-4cb9-8d7a-9e09840b997e None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:44.087891 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c3fd9f-49cd-4cb9-8d7a-9e09840b997e None admin] PATH_INFO: `/v3/projects/service` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:44.093321 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-770a1257-7a69-4c5a-95dd-9c7f5abdb4fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:44.093937 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-770a1257-7a69-4c5a-95dd-9c7f5abdb4fd None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:44.094282 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-770a1257-7a69-4c5a-95dd-9c7f5abdb4fd None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:44.094639 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-770a1257-7a69-4c5a-95dd-9c7f5abdb4fd None admin] PATH_INFO: `/v3/domains` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:44.095365 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-770a1257-7a69-4c5a-95dd-9c7f5abdb4fd None admin] RBAC: Adding query filter params (name=Default) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:44.095725 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-770a1257-7a69-4c5a-95dd-9c7f5abdb4fd None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:44.097159 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-770a1257-7a69-4c5a-95dd-9c7f5abdb4fd None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:44.097808 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4c3fd9f-49cd-4cb9-8d7a-9e09840b997e None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:44.100262 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4c3fd9f-49cd-4cb9-8d7a-9e09840b997e None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:44.114201 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 295/599] 10.222.0.77 () {56 vars in 1109 bytes} [Tue Aug 30 04:59:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:44.114201 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-770a1257-7a69-4c5a-95dd-9c7f5abdb4fd None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:44.120549 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-d4c3fd9f-49cd-4cb9-8d7a-9e09840b997e None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:59:44.120549 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:44.120549 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:44.120549 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:44.120549 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:44.120549 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:44.120549 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:44.120549 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:44.120549 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:44.120549 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:44.120549 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:44.120549 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:44.120549 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 04:59:44.120549 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 04:59:44.120549 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 04:59:44.121623 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 04:59:44.121623 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:44.121623 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:44.121623 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:44.121623 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:44.121623 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:44.121623 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:44.121623 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:44.121623 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:44.121623 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:44.121623 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:44.121623 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:44.121623 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:44.121623 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:44.121623 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:44.121623 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:44.121623 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:44.122710 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 04:59:44.122710 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 04:59:44.122710 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 04:59:44.122710 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 04:59:44.122710 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 04:59:44.122710 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 04:59:44.122710 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:59:44.122710 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 04:59:44.125325 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 305/600] 10.222.0.77 () {56 vars in 1111 bytes} [Tue Aug 30 04:59:43 2022] GET /identity/v3/projects/service => generated 88 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:44.126394 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4c3fd9f-49cd-4cb9-8d7a-9e09840b997e None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:44.153284 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:44.177266 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:44.225307 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-770a1257-7a69-4c5a-95dd-9c7f5abdb4fd None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:44.243921 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4c3fd9f-49cd-4cb9-8d7a-9e09840b997e None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:44.261048 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-589a521b-839e-4469-b9b7-428bd5d4c321 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:44.261680 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-589a521b-839e-4469-b9b7-428bd5d4c321 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:44.261960 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-589a521b-839e-4469-b9b7-428bd5d4c321 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:44.262234 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-589a521b-839e-4469-b9b7-428bd5d4c321 None admin] PATH_INFO: `/v3/projects` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:44.262942 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-589a521b-839e-4469-b9b7-428bd5d4c321 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:44.263249 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-589a521b-839e-4469-b9b7-428bd5d4c321 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:44.264677 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-589a521b-839e-4469-b9b7-428bd5d4c321 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:44.276942 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 296/601] 10.222.0.77 () {56 vars in 1148 bytes} [Tue Aug 30 04:59:44 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:44.279211 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-898d370d-af19-40e8-a003-0ae5f059daab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:44.279837 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-898d370d-af19-40e8-a003-0ae5f059daab None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:44.280114 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-898d370d-af19-40e8-a003-0ae5f059daab None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:44.280382 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-898d370d-af19-40e8-a003-0ae5f059daab None admin] PATH_INFO: `/v3/projects/service` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:44.280754 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-589a521b-839e-4469-b9b7-428bd5d4c321 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:44.287541 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-898d370d-af19-40e8-a003-0ae5f059daab None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:44.289547 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-898d370d-af19-40e8-a003-0ae5f059daab None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:44.301028 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-898d370d-af19-40e8-a003-0ae5f059daab None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:59:44.301028 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:44.301028 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:44.301028 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:44.301028 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:44.301028 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:44.301028 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:44.301028 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:44.301028 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:44.301028 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:44.301028 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:44.301028 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:44.301028 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 04:59:44.301028 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 04:59:44.301028 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 04:59:44.302072 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 04:59:44.302072 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:44.302072 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:44.302072 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:44.302072 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:44.302072 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:44.302072 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:44.302072 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:44.302072 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:44.302072 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:44.302072 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:44.302072 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:44.302072 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:44.302072 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:44.302072 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:44.302072 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:44.302072 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:44.303190 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 04:59:44.303190 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 04:59:44.303190 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 04:59:44.303190 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 04:59:44.303190 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 04:59:44.303190 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 04:59:44.303190 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:59:44.303190 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 04:59:44.303190 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 306/602] 10.222.0.77 () {56 vars in 1111 bytes} [Tue Aug 30 04:59:44 2022] GET /identity/v3/projects/service => generated 88 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:44.303796 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2926f39f-438c-47d5-87b0-321bbbad3da2 None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:44.303796 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2926f39f-438c-47d5-87b0-321bbbad3da2 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:44.303960 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2926f39f-438c-47d5-87b0-321bbbad3da2 None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:44.311277 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 307/603] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:59:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:59:44.311277 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2926f39f-438c-47d5-87b0-321bbbad3da2 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:44.389673 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-589a521b-839e-4469-b9b7-428bd5d4c321 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:44.397681 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:44.404857 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:44.422560 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2926f39f-438c-47d5-87b0-321bbbad3da2 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:44.426831 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa5317b5-1e53-4806-b3d8-74aa013dd860 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:44.427469 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa5317b5-1e53-4806-b3d8-74aa013dd860 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:44.427747 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa5317b5-1e53-4806-b3d8-74aa013dd860 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:44.428015 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa5317b5-1e53-4806-b3d8-74aa013dd860 None admin] PATH_INFO: `/v3/projects` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:44.428706 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa5317b5-1e53-4806-b3d8-74aa013dd860 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:44.429008 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa5317b5-1e53-4806-b3d8-74aa013dd860 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:44.431013 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa5317b5-1e53-4806-b3d8-74aa013dd860 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:44.442849 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 297/604] 10.222.0.77 () {56 vars in 1148 bytes} [Tue Aug 30 04:59:44 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:44.443687 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa5317b5-1e53-4806-b3d8-74aa013dd860 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:44.470488 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ba3a03b-2f77-443e-a093-fa9d633c7ae5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:44.471305 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba3a03b-2f77-443e-a093-fa9d633c7ae5 None admin] REQUEST_METHOD: `PUT` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:44.471528 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba3a03b-2f77-443e-a093-fa9d633c7ae5 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:44.471798 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba3a03b-2f77-443e-a093-fa9d633c7ae5 None admin] PATH_INFO: `/v3/projects/e9919141e3b947ab8d91b55bba4a7de6/users/886eb5e00bbf4ec292eb962b665b37d0/roles/821647e8b40542499667e29e1b301cb2` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:44.487556 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ba3a03b-2f77-443e-a093-fa9d633c7ae5 None admin] RBAC: Authorizing `identity:create_grant(project_id=e9919141e3b947ab8d91b55bba4a7de6, user_id=886eb5e00bbf4ec292eb962b665b37d0, role_id=821647e8b40542499667e29e1b301cb2)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:44.488990 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ba3a03b-2f77-443e-a093-fa9d633c7ae5 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:44.503072 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 308/605] 10.222.0.77 () {58 vars in 1439 bytes} [Tue Aug 30 04:59:44 2022] PUT /identity/v3/projects/e9919141e3b947ab8d91b55bba4a7de6/users/886eb5e00bbf4ec292eb962b665b37d0/roles/821647e8b40542499667e29e1b301cb2 => generated 0 bytes in 197 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:59:44.504295 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8e7a1ab-6ca0-486f-a4b2-31d21cd6a021 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:44.504618 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8e7a1ab-6ca0-486f-a4b2-31d21cd6a021 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:44.504844 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8e7a1ab-6ca0-486f-a4b2-31d21cd6a021 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:44.524550 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-d8e7a1ab-6ca0-486f-a4b2-31d21cd6a021 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:59:44.525771 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-d8e7a1ab-6ca0-486f-a4b2-31d21cd6a021 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:59:44.586736 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa5317b5-1e53-4806-b3d8-74aa013dd860 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:44.613605 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 309/606] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:59:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:59:44.614981 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8e7a1ab-6ca0-486f-a4b2-31d21cd6a021 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:44.627052 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67ced780-9c90-4df1-bd72-80d84ea29314 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:44.627675 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ced780-9c90-4df1-bd72-80d84ea29314 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:44.627952 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ced780-9c90-4df1-bd72-80d84ea29314 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:44.628223 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ced780-9c90-4df1-bd72-80d84ea29314 None admin] PATH_INFO: `/v3/projects` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:44.629258 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67ced780-9c90-4df1-bd72-80d84ea29314 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:44.629383 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67ced780-9c90-4df1-bd72-80d84ea29314 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:44.631101 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67ced780-9c90-4df1-bd72-80d84ea29314 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:44.645206 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 298/607] 10.222.0.77 () {56 vars in 1148 bytes} [Tue Aug 30 04:59:44 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:44.651150 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67ced780-9c90-4df1-bd72-80d84ea29314 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:44.690131 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:44.701130 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:44.744850 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8e7a1ab-6ca0-486f-a4b2-31d21cd6a021 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:44.776606 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67ced780-9c90-4df1-bd72-80d84ea29314 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:44.805755 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a637bde-029b-4039-b1b3-d6bc12b75384 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:44.806415 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a637bde-029b-4039-b1b3-d6bc12b75384 None admin] REQUEST_METHOD: `PUT` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:44.806693 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a637bde-029b-4039-b1b3-d6bc12b75384 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:44.807011 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a637bde-029b-4039-b1b3-d6bc12b75384 None admin] PATH_INFO: `/v3/projects/e9919141e3b947ab8d91b55bba4a7de6/users/019a79b2affe478b8a30d1c538d0415c/roles/821647e8b40542499667e29e1b301cb2` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:44.823183 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a637bde-029b-4039-b1b3-d6bc12b75384 None admin] RBAC: Authorizing `identity:create_grant(project_id=e9919141e3b947ab8d91b55bba4a7de6, user_id=019a79b2affe478b8a30d1c538d0415c, role_id=821647e8b40542499667e29e1b301cb2)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:44.824718 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a637bde-029b-4039-b1b3-d6bc12b75384 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:44.834194 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad07ad9e-21ac-4152-859d-883991fe39be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:44.835059 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad07ad9e-21ac-4152-859d-883991fe39be None admin] REQUEST_METHOD: `PUT` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:44.835449 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad07ad9e-21ac-4152-859d-883991fe39be None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:44.835831 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad07ad9e-21ac-4152-859d-883991fe39be None admin] PATH_INFO: `/v3/projects/e9919141e3b947ab8d91b55bba4a7de6/users/c33c45f4770d4151858e1d897d09af69/roles/821647e8b40542499667e29e1b301cb2` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:44.840018 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 310/608] 10.222.0.77 () {58 vars in 1439 bytes} [Tue Aug 30 04:59:44 2022] PUT /identity/v3/projects/e9919141e3b947ab8d91b55bba4a7de6/users/019a79b2affe478b8a30d1c538d0415c/roles/821647e8b40542499667e29e1b301cb2 => generated 0 bytes in 225 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:59:44.875883 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad07ad9e-21ac-4152-859d-883991fe39be None admin] RBAC: Authorizing `identity:create_grant(project_id=e9919141e3b947ab8d91b55bba4a7de6, user_id=c33c45f4770d4151858e1d897d09af69, role_id=821647e8b40542499667e29e1b301cb2)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:44.877444 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad07ad9e-21ac-4152-859d-883991fe39be None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:44.905602 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 299/609] 10.222.0.77 () {58 vars in 1439 bytes} [Tue Aug 30 04:59:44 2022] PUT /identity/v3/projects/e9919141e3b947ab8d91b55bba4a7de6/users/c33c45f4770d4151858e1d897d09af69/roles/821647e8b40542499667e29e1b301cb2 => generated 0 bytes in 255 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:59:44.955345 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a637bde-029b-4039-b1b3-d6bc12b75384 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:45.087933 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a637bde-029b-4039-b1b3-d6bc12b75384 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:45.098649 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:45.129571 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bdd0253-1119-471b-9920-6965bd544bc8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:45.130218 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bdd0253-1119-471b-9920-6965bd544bc8 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:45.130753 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bdd0253-1119-471b-9920-6965bd544bc8 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:45.131599 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bdd0253-1119-471b-9920-6965bd544bc8 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:45.144186 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bdd0253-1119-471b-9920-6965bd544bc8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:59:45.149314 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-4bdd0253-1119-471b-9920-6965bd544bc8 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 30 04:59:45.149314 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:45.149314 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:45.149314 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:45.149314 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:45.149314 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:45.149314 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:45.149314 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:45.149314 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:45.149314 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:45.149314 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:45.149314 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:45.149314 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:59:45.149314 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:59:45.149314 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:59:45.149314 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:59:45.150225 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:59:45.150225 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:59:45.150225 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:59:45.150225 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:59:45.150225 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:45.150225 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:45.150225 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:45.150225 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:45.150225 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:45.150225 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:45.150225 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:45.150225 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:45.150225 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:45.150225 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:45.150225 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:45.150225 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:45.151083 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:45.151083 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:45.151083 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:45.151083 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:45.151083 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:59:45.151083 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:59:45.151083 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:59:45.151083 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:59:45.151083 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:59:45.151083 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:59:45.151083 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 30 04:59:45.151083 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 04:59:45.151083 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 311/610] 10.222.0.77 () {56 vars in 1102 bytes} [Tue Aug 30 04:59:44 2022] GET /identity/v3/roles/service => generated 85 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:45.156539 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad07ad9e-21ac-4152-859d-883991fe39be None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:45.245747 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:45.262965 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad07ad9e-21ac-4152-859d-883991fe39be None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:45.306564 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0f2e81c-e712-4557-b183-2b1f6ce9fbab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:45.307370 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0f2e81c-e712-4557-b183-2b1f6ce9fbab None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:45.307746 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0f2e81c-e712-4557-b183-2b1f6ce9fbab None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:45.308118 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0f2e81c-e712-4557-b183-2b1f6ce9fbab None admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:45.308886 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0f2e81c-e712-4557-b183-2b1f6ce9fbab None admin] RBAC: Adding query filter params (name=service) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:45.309290 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0f2e81c-e712-4557-b183-2b1f6ce9fbab None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:45.310768 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0f2e81c-e712-4557-b183-2b1f6ce9fbab None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:45.319998 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 300/611] 10.222.0.77 () {56 vars in 1103 bytes} [Tue Aug 30 04:59:45 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:45.328012 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bdd0253-1119-471b-9920-6965bd544bc8 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:45.478471 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bdd0253-1119-471b-9920-6965bd544bc8 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:45.530265 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e6a8873-7ce9-45b6-8a22-e35c66ffdb12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:45.530991 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e6a8873-7ce9-45b6-8a22-e35c66ffdb12 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:45.531270 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e6a8873-7ce9-45b6-8a22-e35c66ffdb12 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:45.531535 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e6a8873-7ce9-45b6-8a22-e35c66ffdb12 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:45.543053 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e6a8873-7ce9-45b6-8a22-e35c66ffdb12 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:45.544789 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e6a8873-7ce9-45b6-8a22-e35c66ffdb12 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:45.559523 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-1e6a8873-7ce9-45b6-8a22-e35c66ffdb12 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:59:45.559523 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:45.559523 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:45.559523 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:45.559523 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:45.559523 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:45.559523 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:45.559523 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:45.559523 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:45.559523 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:45.559523 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:45.559523 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:45.559523 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:59:45.559523 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:59:45.559523 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:59:45.559523 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:59:45.560627 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:45.560627 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:45.560627 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:45.560627 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:45.560627 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:45.560627 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:45.560627 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:45.560627 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:45.560627 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:45.560627 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:45.560627 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:45.560627 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:45.560627 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:45.560627 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:45.560627 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:45.560627 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:45.560627 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:59:45.561697 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:59:45.561697 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:59:45.561697 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 04:59:45.561697 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 312/612] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 04:59:45 2022] GET /identity/v3/domains/Default => generated 87 bytes in 233 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:45.566311 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0f2e81c-e712-4557-b183-2b1f6ce9fbab None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:45.726430 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0f2e81c-e712-4557-b183-2b1f6ce9fbab None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:45.773549 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff9ed2bc-59ad-478e-8914-97578090b81c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:45.774166 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff9ed2bc-59ad-478e-8914-97578090b81c None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:45.774444 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff9ed2bc-59ad-478e-8914-97578090b81c None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:45.774716 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff9ed2bc-59ad-478e-8914-97578090b81c None admin] PATH_INFO: `/v3/domains` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:45.775380 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff9ed2bc-59ad-478e-8914-97578090b81c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:45.775681 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff9ed2bc-59ad-478e-8914-97578090b81c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:45.777060 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff9ed2bc-59ad-478e-8914-97578090b81c None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:45.794076 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 301/613] 10.222.0.77 () {56 vars in 1109 bytes} [Tue Aug 30 04:59:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:45.799432 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e6a8873-7ce9-45b6-8a22-e35c66ffdb12 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:45.842484 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:45.952012 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e6a8873-7ce9-45b6-8a22-e35c66ffdb12 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:45.963892 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c1967d-80bd-4ca4-802b-4723c7b144bd None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:45.964181 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c1967d-80bd-4ca4-802b-4723c7b144bd None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:45.964457 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c1967d-80bd-4ca4-802b-4723c7b144bd None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:45.966061 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 302/614] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:59:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:59:45.984151 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba9a5b84-0726-4cc4-8a13-4b43120b3a94 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:45.984389 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba9a5b84-0726-4cc4-8a13-4b43120b3a94 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:45.984641 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba9a5b84-0726-4cc4-8a13-4b43120b3a94 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:45.989680 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14f397a1-777c-46d6-919b-7bda017fa88d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:45.990311 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14f397a1-777c-46d6-919b-7bda017fa88d None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:45.990592 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14f397a1-777c-46d6-919b-7bda017fa88d None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:45.990887 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14f397a1-777c-46d6-919b-7bda017fa88d None admin] PATH_INFO: `/v3/users/swift` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:45.998799 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14f397a1-777c-46d6-919b-7bda017fa88d None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:46.000251 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14f397a1-777c-46d6-919b-7bda017fa88d None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:46.005533 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:46.012399 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-14f397a1-777c-46d6-919b-7bda017fa88d None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Aug 30 04:59:46.012399 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:46.012399 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:46.012399 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:46.012399 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:46.012399 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:46.012399 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:46.012399 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:46.012399 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:46.012399 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:46.012399 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:46.012399 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:46.012399 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 04:59:46.012399 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 04:59:46.012399 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 04:59:46.012399 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 04:59:46.013307 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:46.013307 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:46.013307 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 04:59:46.013307 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:59:46.013307 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 04:59:46.013307 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:59:46.013307 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:46.013307 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:46.013307 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:46.013307 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:46.013307 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:46.013307 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:46.013307 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:46.013307 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:46.013307 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:46.013307 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:46.013307 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:46.013307 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:46.014219 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:46.014219 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:46.014219 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 04:59:46.014219 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 04:59:46.014219 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 04:59:46.014219 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 04:59:46.014219 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 04:59:46.014219 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 04:59:46.014219 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Aug 30 04:59:46.014219 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 04:59:46.018115 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 313/615] 10.222.0.77 () {56 vars in 1096 bytes} [Tue Aug 30 04:59:45 2022] GET /identity/v3/users/swift => generated 83 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:46.019246 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-ba9a5b84-0726-4cc4-8a13-4b43120b3a94 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:59:46.019359 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14f397a1-777c-46d6-919b-7bda017fa88d None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:46.020547 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-ba9a5b84-0726-4cc4-8a13-4b43120b3a94 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:59:46.104908 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:46.126405 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14f397a1-777c-46d6-919b-7bda017fa88d None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:46.162332 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69555fae-c954-4ff9-8272-5f1c62300cfe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:46.162987 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69555fae-c954-4ff9-8272-5f1c62300cfe None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:46.163270 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69555fae-c954-4ff9-8272-5f1c62300cfe None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:46.163597 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69555fae-c954-4ff9-8272-5f1c62300cfe None admin] PATH_INFO: `/v3/users` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:46.164318 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69555fae-c954-4ff9-8272-5f1c62300cfe None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:46.164623 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69555fae-c954-4ff9-8272-5f1c62300cfe None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:46.166066 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69555fae-c954-4ff9-8272-5f1c62300cfe None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:46.189336 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 314/616] 10.222.0.77 () {56 vars in 1135 bytes} [Tue Aug 30 04:59:46 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:59:46.190625 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69555fae-c954-4ff9-8272-5f1c62300cfe None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:46.221384 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 303/617] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:59:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 238 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:59:46.300073 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69555fae-c954-4ff9-8272-5f1c62300cfe None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:46.338744 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bfcd510-8968-4ae9-9200-39058eda31f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:46.339396 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bfcd510-8968-4ae9-9200-39058eda31f8 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:46.339676 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bfcd510-8968-4ae9-9200-39058eda31f8 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:46.339949 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bfcd510-8968-4ae9-9200-39058eda31f8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:46.346707 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:46.350858 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bfcd510-8968-4ae9-9200-39058eda31f8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:46.352385 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bfcd510-8968-4ae9-9200-39058eda31f8 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:46.368276 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-6bfcd510-8968-4ae9-9200-39058eda31f8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:59:46.368276 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:46.368276 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:46.368276 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:46.368276 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:46.368276 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:46.368276 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:46.368276 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:46.368276 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:46.368276 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:46.368276 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:46.368276 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:46.368276 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:59:46.368276 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:59:46.368276 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:59:46.368276 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:59:46.369315 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:46.369315 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:46.369315 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:46.369315 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:46.369315 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:46.369315 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:46.369315 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:46.369315 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:46.369315 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:46.369315 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:46.369315 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:46.369315 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:46.369315 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:46.369315 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:46.369315 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:46.369315 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:46.369315 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:59:46.370184 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:59:46.370184 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:59:46.370184 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 04:59:46.374122 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 315/618] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 04:59:46 2022] GET /identity/v3/domains/Default => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:46.375502 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba9a5b84-0726-4cc4-8a13-4b43120b3a94 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:46.534530 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba9a5b84-0726-4cc4-8a13-4b43120b3a94 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:46.579996 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-887517c5-215e-4c2a-bc50-f318ed63b43c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:46.580520 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-887517c5-215e-4c2a-bc50-f318ed63b43c None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:46.580803 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-887517c5-215e-4c2a-bc50-f318ed63b43c None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:46.581074 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-887517c5-215e-4c2a-bc50-f318ed63b43c None admin] PATH_INFO: `/v3/domains` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:46.581712 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-887517c5-215e-4c2a-bc50-f318ed63b43c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:46.582013 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-887517c5-215e-4c2a-bc50-f318ed63b43c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:46.583409 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-887517c5-215e-4c2a-bc50-f318ed63b43c None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:46.585854 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:46.600741 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 304/619] 10.222.0.77 () {56 vars in 1109 bytes} [Tue Aug 30 04:59:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:46.602576 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bfcd510-8968-4ae9-9200-39058eda31f8 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:46.613212 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-887517c5-215e-4c2a-bc50-f318ed63b43c None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:46.768868 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-887517c5-215e-4c2a-bc50-f318ed63b43c None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:46.813786 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f37a7a03-1ef1-453e-9ded-2d8b6503970c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:46.814428 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f37a7a03-1ef1-453e-9ded-2d8b6503970c None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:46.814707 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f37a7a03-1ef1-453e-9ded-2d8b6503970c None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:46.815004 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f37a7a03-1ef1-453e-9ded-2d8b6503970c None admin] PATH_INFO: `/v3/projects/service` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:46.829564 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bfcd510-8968-4ae9-9200-39058eda31f8 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:46.833538 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f37a7a03-1ef1-453e-9ded-2d8b6503970c None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:46.834970 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f37a7a03-1ef1-453e-9ded-2d8b6503970c None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:46.855749 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-f37a7a03-1ef1-453e-9ded-2d8b6503970c None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:59:46.855749 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:46.855749 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:46.855749 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:46.855749 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:46.855749 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:46.855749 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:46.855749 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:46.855749 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:46.855749 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:46.855749 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:46.855749 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:46.855749 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 04:59:46.855749 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 04:59:46.855749 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 04:59:46.856843 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 04:59:46.856843 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:46.856843 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:46.856843 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:46.856843 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:46.856843 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:46.856843 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:46.856843 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:46.856843 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:46.856843 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:46.856843 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:46.856843 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:46.856843 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:46.856843 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:46.856843 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:46.856843 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:46.856843 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:46.857727 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 04:59:46.857727 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 04:59:46.857727 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 04:59:46.857727 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 04:59:46.857727 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 04:59:46.857727 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 04:59:46.857727 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:59:46.857727 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 04:59:46.857727 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 305/620] 10.222.0.77 () {56 vars in 1111 bytes} [Tue Aug 30 04:59:46 2022] GET /identity/v3/projects/service => generated 88 bytes in 253 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:46.868041 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f37a7a03-1ef1-453e-9ded-2d8b6503970c None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:46.895346 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4373a3c-db9e-4a6d-9b9c-17c413c25e23 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:46.895977 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4373a3c-db9e-4a6d-9b9c-17c413c25e23 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:46.896253 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4373a3c-db9e-4a6d-9b9c-17c413c25e23 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:46.896518 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4373a3c-db9e-4a6d-9b9c-17c413c25e23 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:46.920895 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4373a3c-db9e-4a6d-9b9c-17c413c25e23 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:59:46.939642 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-c4373a3c-db9e-4a6d-9b9c-17c413c25e23 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 30 04:59:46.939642 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:46.939642 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:46.939642 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:46.939642 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:46.939642 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:46.939642 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:46.939642 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:46.939642 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:46.939642 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:46.939642 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:46.939642 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:46.939642 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:59:46.939642 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:59:46.939642 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:59:46.939642 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:59:46.940626 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:59:46.940626 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:59:46.940626 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:59:46.940626 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:59:46.940626 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:46.940626 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:46.940626 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:46.940626 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:46.940626 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:46.940626 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:46.940626 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:46.940626 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:46.940626 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:46.940626 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:46.940626 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:46.940626 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:46.941471 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:46.941471 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:46.941471 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:46.941471 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:46.941471 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:59:46.941471 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:59:46.941471 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:59:46.941471 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:59:46.941471 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:59:46.941471 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:59:46.941471 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 30 04:59:46.941471 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 04:59:46.941471 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 316/621] 10.222.0.77 () {56 vars in 1102 bytes} [Tue Aug 30 04:59:46 2022] GET /identity/v3/roles/service => generated 85 bytes in 339 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:46.946291 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:46.946419 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4373a3c-db9e-4a6d-9b9c-17c413c25e23 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:47.014863 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:47.021820 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f37a7a03-1ef1-453e-9ded-2d8b6503970c None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:47.077500 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-738d70fe-a73a-4071-8cc1-529235ba69e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:47.078112 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738d70fe-a73a-4071-8cc1-529235ba69e5 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:47.078386 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738d70fe-a73a-4071-8cc1-529235ba69e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:47.078650 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738d70fe-a73a-4071-8cc1-529235ba69e5 None admin] PATH_INFO: `/v3/projects` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:47.079375 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-738d70fe-a73a-4071-8cc1-529235ba69e5 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:47.079672 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-738d70fe-a73a-4071-8cc1-529235ba69e5 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:47.079885 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4373a3c-db9e-4a6d-9b9c-17c413c25e23 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:47.081087 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-738d70fe-a73a-4071-8cc1-529235ba69e5 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:47.094586 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 306/622] 10.222.0.77 () {56 vars in 1148 bytes} [Tue Aug 30 04:59:46 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:47.095600 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be3e3b0-51ca-49e9-a5a7-94ec7c8684ff None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:47.095880 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be3e3b0-51ca-49e9-a5a7-94ec7c8684ff None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:47.096150 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be3e3b0-51ca-49e9-a5a7-94ec7c8684ff None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:47.097497 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 307/623] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:59:47 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:59:47.098302 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48e1e663-bffd-4e73-bc51-a4dec20c75fa None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:47.098568 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48e1e663-bffd-4e73-bc51-a4dec20c75fa None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:47.098849 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48e1e663-bffd-4e73-bc51-a4dec20c75fa None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:47.100373 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 308/624] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:59:47 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:59:47.100908 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48e1e663-bffd-4e73-bc51-a4dec20c75fa None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:47.123625 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0123d7a9-f8be-4566-997d-937d9a4c5f57 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:47.125161 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0123d7a9-f8be-4566-997d-937d9a4c5f57 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:47.125723 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0123d7a9-f8be-4566-997d-937d9a4c5f57 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:47.126193 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0123d7a9-f8be-4566-997d-937d9a4c5f57 None admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:47.127638 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0123d7a9-f8be-4566-997d-937d9a4c5f57 None admin] RBAC: Adding query filter params (name=service) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:47.128114 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0123d7a9-f8be-4566-997d-937d9a4c5f57 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:47.130358 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0123d7a9-f8be-4566-997d-937d9a4c5f57 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:47.139484 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 317/625] 10.222.0.77 () {56 vars in 1103 bytes} [Tue Aug 30 04:59:46 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:59:47.145043 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25c00634-e714-4589-abbb-b95e408cb209 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:47.145333 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25c00634-e714-4589-abbb-b95e408cb209 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:47.145831 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25c00634-e714-4589-abbb-b95e408cb209 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:47.163672 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-25c00634-e714-4589-abbb-b95e408cb209 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:59:47.164910 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-25c00634-e714-4589-abbb-b95e408cb209 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:59:47.206331 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48e1e663-bffd-4e73-bc51-a4dec20c75fa None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:47.222966 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:47.242023 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76a68065-f7b4-41a4-bd2a-55d26c27610a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:47.242023 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a68065-f7b4-41a4-bd2a-55d26c27610a None admin] REQUEST_METHOD: `PUT` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:47.242023 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a68065-f7b4-41a4-bd2a-55d26c27610a None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:47.242364 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a68065-f7b4-41a4-bd2a-55d26c27610a None admin] PATH_INFO: `/v3/projects/e9919141e3b947ab8d91b55bba4a7de6/users/e0a7f7afc3d44ec788c05d005e52a1b8/roles/821647e8b40542499667e29e1b301cb2` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:47.243151 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 318/626] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:59:47 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:59:47.244765 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-748b6a1f-b831-4c37-adbf-a7add49baf8a None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:47.245259 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-748b6a1f-b831-4c37-adbf-a7add49baf8a None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:47.245540 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-748b6a1f-b831-4c37-adbf-a7add49baf8a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:47.261516 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76a68065-f7b4-41a4-bd2a-55d26c27610a None admin] RBAC: Authorizing `identity:create_grant(project_id=e9919141e3b947ab8d91b55bba4a7de6, user_id=e0a7f7afc3d44ec788c05d005e52a1b8, role_id=821647e8b40542499667e29e1b301cb2)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:47.262998 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76a68065-f7b4-41a4-bd2a-55d26c27610a None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:47.265099 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-748b6a1f-b831-4c37-adbf-a7add49baf8a None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:59:47.266504 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-748b6a1f-b831-4c37-adbf-a7add49baf8a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:59:47.279682 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 309/627] 10.222.0.77 () {58 vars in 1439 bytes} [Tue Aug 30 04:59:47 2022] PUT /identity/v3/projects/e9919141e3b947ab8d91b55bba4a7de6/users/e0a7f7afc3d44ec788c05d005e52a1b8/roles/821647e8b40542499667e29e1b301cb2 => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:59:47.280882 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76a68065-f7b4-41a4-bd2a-55d26c27610a None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:47.308595 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:47.354919 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 319/628] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:59:47 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:59:47.354919 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4abae33-a6df-4f68-8051-c0b7d15cc10b None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:47.354919 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4abae33-a6df-4f68-8051-c0b7d15cc10b None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:47.354919 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4abae33-a6df-4f68-8051-c0b7d15cc10b None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:47.354919 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 320/629] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:59:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:59:47.356862 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ef7ba6b-543e-4ec4-8713-f5ec9a59f74b None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:47.357141 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ef7ba6b-543e-4ec4-8713-f5ec9a59f74b None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:47.357410 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ef7ba6b-543e-4ec4-8713-f5ec9a59f74b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:47.396679 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-2ef7ba6b-543e-4ec4-8713-f5ec9a59f74b None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:59:47.397888 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-2ef7ba6b-543e-4ec4-8713-f5ec9a59f74b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:59:47.475727 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76a68065-f7b4-41a4-bd2a-55d26c27610a None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:47.497225 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 321/630] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:59:47 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:59:47.514703 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6da993ef-b833-4f08-bfe3-8a73a0e6d5e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:47.515376 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da993ef-b833-4f08-bfe3-8a73a0e6d5e1 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:47.515656 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da993ef-b833-4f08-bfe3-8a73a0e6d5e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:47.515926 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da993ef-b833-4f08-bfe3-8a73a0e6d5e1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:47.530689 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6da993ef-b833-4f08-bfe3-8a73a0e6d5e1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:47.532510 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6da993ef-b833-4f08-bfe3-8a73a0e6d5e1 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:47.548052 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-6da993ef-b833-4f08-bfe3-8a73a0e6d5e1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:59:47.548052 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:47.548052 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:47.548052 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:47.548052 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:47.548052 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:47.548052 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:47.548052 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:47.548052 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:47.548052 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:47.548052 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:47.548052 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:47.548052 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:59:47.548052 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:59:47.548052 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:59:47.548052 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:59:47.549116 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:47.549116 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:47.549116 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:47.549116 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:47.549116 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:47.549116 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:47.549116 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:47.549116 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:47.549116 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:47.549116 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:47.549116 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:47.549116 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:47.549116 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:47.549116 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:47.549116 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:47.549116 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:47.549116 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:59:47.550024 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:59:47.550024 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:59:47.550024 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 04:59:47.552120 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 310/631] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 04:59:47 2022] GET /identity/v3/domains/Default => generated 87 bytes in 272 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:47.558022 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ef7ba6b-543e-4ec4-8713-f5ec9a59f74b None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:47.568938 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6da993ef-b833-4f08-bfe3-8a73a0e6d5e1 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:47.608668 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:47.720663 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ef7ba6b-543e-4ec4-8713-f5ec9a59f74b None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:47.720900 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6da993ef-b833-4f08-bfe3-8a73a0e6d5e1 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:47.722333 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:47.763936 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b4045f4-bc1f-4629-9b75-281419c3e752 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:47.764571 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b4045f4-bc1f-4629-9b75-281419c3e752 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:47.764850 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b4045f4-bc1f-4629-9b75-281419c3e752 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:47.765136 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b4045f4-bc1f-4629-9b75-281419c3e752 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:47.765549 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-311c360c-711d-4d32-9fb6-e434ac0b4d1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:47.766218 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-311c360c-711d-4d32-9fb6-e434ac0b4d1e None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:47.766560 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-311c360c-711d-4d32-9fb6-e434ac0b4d1e None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:47.766946 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-311c360c-711d-4d32-9fb6-e434ac0b4d1e None admin] PATH_INFO: `/v3/domains` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:47.767901 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-311c360c-711d-4d32-9fb6-e434ac0b4d1e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:47.768622 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-311c360c-711d-4d32-9fb6-e434ac0b4d1e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:47.770049 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-311c360c-711d-4d32-9fb6-e434ac0b4d1e None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:47.777665 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b4045f4-bc1f-4629-9b75-281419c3e752 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:59:47.783105 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 322/632] 10.222.0.77 () {56 vars in 1109 bytes} [Tue Aug 30 04:59:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:47.783252 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-3b4045f4-bc1f-4629-9b75-281419c3e752 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 30 04:59:47.783252 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:47.783252 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:47.783252 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:47.783252 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:47.783252 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:47.783252 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:47.783252 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:47.783252 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:47.783252 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:47.783252 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:47.783252 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:47.783252 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:59:47.783252 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:59:47.783252 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:59:47.783252 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:59:47.784140 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:59:47.784140 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:59:47.784140 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:59:47.784140 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:59:47.784140 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:47.784140 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:47.784140 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:47.784140 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:47.784140 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:47.784140 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:47.784140 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:47.784140 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:47.784140 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:47.784140 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:47.784140 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:47.784140 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:47.784994 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:47.784994 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:47.784994 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:47.784994 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:47.784994 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:59:47.784994 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:59:47.784994 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:59:47.784994 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:59:47.784994 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:59:47.784994 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:59:47.784994 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 30 04:59:47.784994 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 04:59:47.787026 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-311c360c-711d-4d32-9fb6-e434ac0b4d1e None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:47.789396 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 311/633] 10.222.0.77 () {56 vars in 1102 bytes} [Tue Aug 30 04:59:47 2022] GET /identity/v3/roles/service => generated 85 bytes in 225 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:47.790478 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b4045f4-bc1f-4629-9b75-281419c3e752 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:47.880930 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:47.892554 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-311c360c-711d-4d32-9fb6-e434ac0b4d1e None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:47.906688 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b4045f4-bc1f-4629-9b75-281419c3e752 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:47.930367 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d069c18c-ffda-4807-a73d-8f60cd7cea77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:47.931100 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d069c18c-ffda-4807-a73d-8f60cd7cea77 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:47.931386 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d069c18c-ffda-4807-a73d-8f60cd7cea77 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:47.931652 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d069c18c-ffda-4807-a73d-8f60cd7cea77 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:47.941931 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d069c18c-ffda-4807-a73d-8f60cd7cea77 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:59:47.943268 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb83eff3-24cd-4560-ad1a-2fa9a05b5444 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:47.943878 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb83eff3-24cd-4560-ad1a-2fa9a05b5444 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:47.944153 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb83eff3-24cd-4560-ad1a-2fa9a05b5444 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:47.944419 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb83eff3-24cd-4560-ad1a-2fa9a05b5444 None admin] PATH_INFO: `/v3/users/glance` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:47.947382 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-d069c18c-ffda-4807-a73d-8f60cd7cea77 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 30 04:59:47.947382 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:47.947382 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:47.947382 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:47.947382 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:47.947382 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:47.947382 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:47.947382 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:47.947382 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:47.947382 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:47.947382 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:47.947382 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:47.947382 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:59:47.947382 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:59:47.947382 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:59:47.947382 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:59:47.948316 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:59:47.948316 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:59:47.948316 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:59:47.948316 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:59:47.948316 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:47.948316 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:47.948316 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:47.948316 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:47.948316 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:47.948316 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:47.948316 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:47.948316 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:47.948316 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:47.948316 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:47.948316 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:47.948316 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:47.949186 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:47.949186 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:47.949186 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:47.949186 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:47.949186 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:59:47.949186 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:59:47.949186 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:59:47.949186 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:59:47.949186 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:59:47.949186 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:59:47.949186 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 30 04:59:47.949186 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 04:59:47.949186 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 323/634] 10.222.0.77 () {56 vars in 1102 bytes} [Tue Aug 30 04:59:47 2022] GET /identity/v3/roles/service => generated 85 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:47.949186 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d069c18c-ffda-4807-a73d-8f60cd7cea77 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:47.957889 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb83eff3-24cd-4560-ad1a-2fa9a05b5444 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:47.959520 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb83eff3-24cd-4560-ad1a-2fa9a05b5444 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:47.972166 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-fb83eff3-24cd-4560-ad1a-2fa9a05b5444 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Aug 30 04:59:47.972166 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:47.972166 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:47.972166 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:47.972166 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:47.972166 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:47.972166 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:47.972166 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:47.972166 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:47.972166 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:47.972166 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:47.972166 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:47.972166 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 04:59:47.972166 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 04:59:47.972166 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 04:59:47.972166 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 04:59:47.973115 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:47.973115 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:47.973115 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 04:59:47.973115 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:59:47.973115 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 04:59:47.973115 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:59:47.973115 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:47.973115 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:47.973115 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:47.973115 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:47.973115 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:47.973115 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:47.973115 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:47.973115 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:47.973115 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:47.973115 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:47.973115 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:47.973115 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:47.974043 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:47.974043 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:47.974043 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 04:59:47.974043 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 04:59:47.974043 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 04:59:47.974043 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 04:59:47.974043 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 04:59:47.974043 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 04:59:47.974043 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Aug 30 04:59:47.974043 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 04:59:47.978580 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:47.978715 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 312/635] 10.222.0.77 () {56 vars in 1099 bytes} [Tue Aug 30 04:59:47 2022] GET /identity/v3/users/glance => generated 84 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:47.979478 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb83eff3-24cd-4560-ad1a-2fa9a05b5444 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:48.060483 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d069c18c-ffda-4807-a73d-8f60cd7cea77 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:48.081498 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb83eff3-24cd-4560-ad1a-2fa9a05b5444 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:48.096404 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2fc5c99-8ee3-47a9-a5f6-24805aa40776 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:48.097004 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2fc5c99-8ee3-47a9-a5f6-24805aa40776 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:48.097288 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2fc5c99-8ee3-47a9-a5f6-24805aa40776 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:48.097558 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2fc5c99-8ee3-47a9-a5f6-24805aa40776 None admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:48.098217 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2fc5c99-8ee3-47a9-a5f6-24805aa40776 None admin] RBAC: Adding query filter params (name=service) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:48.098519 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2fc5c99-8ee3-47a9-a5f6-24805aa40776 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:48.099915 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2fc5c99-8ee3-47a9-a5f6-24805aa40776 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:48.113340 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 324/636] 10.222.0.77 () {56 vars in 1103 bytes} [Tue Aug 30 04:59:47 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:48.113340 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2fc5c99-8ee3-47a9-a5f6-24805aa40776 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:48.121698 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2b9c613-c698-4c08-bf39-72b76f942cfc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:48.122318 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2b9c613-c698-4c08-bf39-72b76f942cfc None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:48.122597 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2b9c613-c698-4c08-bf39-72b76f942cfc None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:48.122894 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2b9c613-c698-4c08-bf39-72b76f942cfc None admin] PATH_INFO: `/v3/roles/service` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:48.130259 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:48.137503 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2b9c613-c698-4c08-bf39-72b76f942cfc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:59:48.142667 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-e2b9c613-c698-4c08-bf39-72b76f942cfc None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 30 04:59:48.142667 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:48.142667 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:48.142667 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:48.142667 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:48.142667 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:48.142667 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:48.142667 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:48.142667 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:48.142667 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:48.142667 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:48.142667 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:48.142667 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:59:48.142667 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:59:48.142667 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:59:48.142667 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:59:48.143613 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:59:48.143613 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:59:48.143613 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:59:48.143613 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:59:48.143613 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:48.143613 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:48.143613 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:48.143613 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:48.143613 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:48.143613 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:48.143613 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:48.143613 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:48.143613 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:48.143613 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:48.143613 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:48.143613 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:48.144443 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:48.144443 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:48.144443 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:48.144443 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:48.144443 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:59:48.144443 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:59:48.144443 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:59:48.144443 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:59:48.144443 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:59:48.144443 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:59:48.144443 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 30 04:59:48.144443 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 04:59:48.144443 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 313/637] 10.222.0.77 () {56 vars in 1102 bytes} [Tue Aug 30 04:59:47 2022] GET /identity/v3/roles/service => generated 85 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:48.145135 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2b9c613-c698-4c08-bf39-72b76f942cfc None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:48.204638 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:48.222500 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2fc5c99-8ee3-47a9-a5f6-24805aa40776 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:48.256390 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2b9c613-c698-4c08-bf39-72b76f942cfc None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:48.260422 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f7fe8e7-45ee-458b-990e-8760041ceb09 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:48.261026 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f7fe8e7-45ee-458b-990e-8760041ceb09 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:48.261306 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f7fe8e7-45ee-458b-990e-8760041ceb09 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:48.261579 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f7fe8e7-45ee-458b-990e-8760041ceb09 None admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:48.262229 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f7fe8e7-45ee-458b-990e-8760041ceb09 None admin] RBAC: Adding query filter params (name=service) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:48.262530 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f7fe8e7-45ee-458b-990e-8760041ceb09 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:48.264340 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f7fe8e7-45ee-458b-990e-8760041ceb09 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:48.277923 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 325/638] 10.222.0.77 () {56 vars in 1103 bytes} [Tue Aug 30 04:59:48 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:48.277923 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f7fe8e7-45ee-458b-990e-8760041ceb09 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:48.296937 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e7377b5-8a25-44ab-b920-ee827887f91d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:48.297539 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e7377b5-8a25-44ab-b920-ee827887f91d None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:48.297815 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e7377b5-8a25-44ab-b920-ee827887f91d None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:48.298082 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e7377b5-8a25-44ab-b920-ee827887f91d None admin] PATH_INFO: `/v3/users` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:48.298800 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e7377b5-8a25-44ab-b920-ee827887f91d None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:48.299126 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e7377b5-8a25-44ab-b920-ee827887f91d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:48.300539 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e7377b5-8a25-44ab-b920-ee827887f91d None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:48.318332 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 314/639] 10.222.0.77 () {56 vars in 1137 bytes} [Tue Aug 30 04:59:48 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:48.320776 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e7377b5-8a25-44ab-b920-ee827887f91d None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:48.380344 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:48.385970 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f7fe8e7-45ee-458b-990e-8760041ceb09 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:48.425535 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11a9cb2a-0cd7-40a6-9cfe-23ae80252385 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:48.426186 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11a9cb2a-0cd7-40a6-9cfe-23ae80252385 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:48.426462 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11a9cb2a-0cd7-40a6-9cfe-23ae80252385 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:48.426725 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11a9cb2a-0cd7-40a6-9cfe-23ae80252385 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:48.427962 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e7377b5-8a25-44ab-b920-ee827887f91d None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:48.442384 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11a9cb2a-0cd7-40a6-9cfe-23ae80252385 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:48.443900 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11a9cb2a-0cd7-40a6-9cfe-23ae80252385 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:48.452000 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:48.462864 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-11a9cb2a-0cd7-40a6-9cfe-23ae80252385 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:59:48.462864 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:48.462864 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:48.462864 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:48.462864 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:48.462864 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:48.462864 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:48.462864 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:48.462864 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:48.462864 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:48.462864 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:48.462864 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:48.462864 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:59:48.462864 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:59:48.462864 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:59:48.462864 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:59:48.463851 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:48.463851 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:48.463851 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:48.463851 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:48.463851 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:48.463851 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:48.463851 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:48.463851 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:48.463851 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:48.463851 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:48.463851 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:48.463851 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:48.463851 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:48.463851 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:48.463851 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:48.463851 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:48.463851 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:59:48.464856 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:59:48.464856 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:59:48.464856 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 04:59:48.468710 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 326/640] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 04:59:48 2022] GET /identity/v3/domains/Default => generated 87 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:48.469552 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11a9cb2a-0cd7-40a6-9cfe-23ae80252385 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:48.474920 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a64c3981-1ed5-4435-988f-e06969ed70d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:48.475559 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a64c3981-1ed5-4435-988f-e06969ed70d9 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:48.475841 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a64c3981-1ed5-4435-988f-e06969ed70d9 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:48.476352 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a64c3981-1ed5-4435-988f-e06969ed70d9 None admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:48.476804 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a64c3981-1ed5-4435-988f-e06969ed70d9 None admin] RBAC: Adding query filter params (name=service) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:48.477104 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a64c3981-1ed5-4435-988f-e06969ed70d9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:48.479172 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a64c3981-1ed5-4435-988f-e06969ed70d9 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:48.487918 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 315/641] 10.222.0.77 () {56 vars in 1103 bytes} [Tue Aug 30 04:59:48 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:48.488705 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a64c3981-1ed5-4435-988f-e06969ed70d9 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:48.584816 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11a9cb2a-0cd7-40a6-9cfe-23ae80252385 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:48.593154 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a64c3981-1ed5-4435-988f-e06969ed70d9 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:48.610597 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:48.625885 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c35203ae-059d-48ce-b21d-eee7bce96ae6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:48.626499 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c35203ae-059d-48ce-b21d-eee7bce96ae6 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:48.626768 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c35203ae-059d-48ce-b21d-eee7bce96ae6 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:48.627061 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c35203ae-059d-48ce-b21d-eee7bce96ae6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:48.627858 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f14dcc8b-7554-4c3a-b5ab-78eeba275ac4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:48.628458 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f14dcc8b-7554-4c3a-b5ab-78eeba275ac4 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:48.628746 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f14dcc8b-7554-4c3a-b5ab-78eeba275ac4 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:48.629021 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f14dcc8b-7554-4c3a-b5ab-78eeba275ac4 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:48.638606 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c35203ae-059d-48ce-b21d-eee7bce96ae6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:48.640280 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c35203ae-059d-48ce-b21d-eee7bce96ae6 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:48.657048 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-c35203ae-059d-48ce-b21d-eee7bce96ae6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:59:48.657048 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:48.657048 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:48.657048 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:48.657048 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:48.657048 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:48.657048 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:48.657048 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:48.657048 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:48.657048 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:48.657048 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:48.657048 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:48.657048 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:59:48.657048 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:59:48.657048 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:59:48.657048 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:59:48.658032 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:48.658032 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:48.658032 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:48.658032 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:48.658032 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:48.658032 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:48.658032 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:48.658032 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:48.658032 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:48.658032 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:48.658032 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:48.658032 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:48.658032 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:48.658032 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:48.658032 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:48.658032 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:48.658032 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:59:48.659494 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:59:48.659494 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:59:48.659494 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 04:59:48.659494 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 327/642] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 04:59:48 2022] GET /identity/v3/domains/Default => generated 87 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:48.659754 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c35203ae-059d-48ce-b21d-eee7bce96ae6 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:48.710012 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f14dcc8b-7554-4c3a-b5ab-78eeba275ac4 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:48.711555 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f14dcc8b-7554-4c3a-b5ab-78eeba275ac4 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:48.711695 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:48.765735 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c35203ae-059d-48ce-b21d-eee7bce96ae6 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:48.802612 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-280c92c1-5a7f-4225-85f4-eb50eb32ca11 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:48.803334 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-280c92c1-5a7f-4225-85f4-eb50eb32ca11 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:48.803629 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-280c92c1-5a7f-4225-85f4-eb50eb32ca11 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:48.803900 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-280c92c1-5a7f-4225-85f4-eb50eb32ca11 None admin] PATH_INFO: `/v3/domains` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:48.804560 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-280c92c1-5a7f-4225-85f4-eb50eb32ca11 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:48.804857 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-280c92c1-5a7f-4225-85f4-eb50eb32ca11 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:48.806324 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-280c92c1-5a7f-4225-85f4-eb50eb32ca11 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:48.821725 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 316/643] 10.222.0.77 () {58 vars in 1282 bytes} [Tue Aug 30 04:59:48 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 329 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 04:59:48.821725 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f14dcc8b-7554-4c3a-b5ab-78eeba275ac4 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:48.821983 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 328/644] 10.222.0.77 () {56 vars in 1109 bytes} [Tue Aug 30 04:59:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:48.821983 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-280c92c1-5a7f-4225-85f4-eb50eb32ca11 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:48.841170 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:48.930721 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-280c92c1-5a7f-4225-85f4-eb50eb32ca11 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:48.932395 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f14dcc8b-7554-4c3a-b5ab-78eeba275ac4 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:48.941768 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:48.969727 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbd69b7e-a0c4-42d5-947e-fbec5bf97320 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:48.970226 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd69b7e-a0c4-42d5-947e-fbec5bf97320 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:48.970506 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd69b7e-a0c4-42d5-947e-fbec5bf97320 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:48.970779 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd69b7e-a0c4-42d5-947e-fbec5bf97320 None admin] PATH_INFO: `/v3/domains` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:48.971442 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbd69b7e-a0c4-42d5-947e-fbec5bf97320 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:48.971749 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbd69b7e-a0c4-42d5-947e-fbec5bf97320 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:48.973109 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbd69b7e-a0c4-42d5-947e-fbec5bf97320 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:48.977047 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8a7f659-d059-49d0-a3d3-6c008ba79fa5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:48.977896 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a7f659-d059-49d0-a3d3-6c008ba79fa5 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:48.978633 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a7f659-d059-49d0-a3d3-6c008ba79fa5 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:48.979140 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a7f659-d059-49d0-a3d3-6c008ba79fa5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:48.985973 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 329/645] 10.222.0.77 () {56 vars in 1109 bytes} [Tue Aug 30 04:59:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:48.986753 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbd69b7e-a0c4-42d5-947e-fbec5bf97320 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:48.995341 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8a7f659-d059-49d0-a3d3-6c008ba79fa5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:48.996803 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8a7f659-d059-49d0-a3d3-6c008ba79fa5 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:49.014008 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-b8a7f659-d059-49d0-a3d3-6c008ba79fa5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:59:49.014008 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:49.014008 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:49.014008 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:49.014008 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:49.014008 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:49.014008 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:49.014008 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:49.014008 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:49.014008 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:49.014008 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:49.014008 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:49.014008 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:59:49.014008 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:59:49.014008 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:59:49.014008 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:59:49.015056 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:49.015056 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:49.015056 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:49.015056 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:49.015056 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:49.015056 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:49.015056 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:49.015056 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:49.015056 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:49.015056 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:49.015056 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:49.015056 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:49.015056 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:49.015056 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:49.015056 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:49.015056 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:49.015056 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:59:49.016018 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:59:49.016018 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:59:49.016018 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 04:59:49.016018 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 317/646] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 04:59:48 2022] GET /identity/v3/domains/Default => generated 87 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:49.016018 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8a7f659-d059-49d0-a3d3-6c008ba79fa5 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:49.094395 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbd69b7e-a0c4-42d5-947e-fbec5bf97320 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:49.095218 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:49.126800 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8a7f659-d059-49d0-a3d3-6c008ba79fa5 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:49.133259 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3559b043-776e-494f-be31-0bd196b6538b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:49.133880 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3559b043-776e-494f-be31-0bd196b6538b None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:49.134144 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3559b043-776e-494f-be31-0bd196b6538b None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:49.134831 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3559b043-776e-494f-be31-0bd196b6538b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:49.146968 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3559b043-776e-494f-be31-0bd196b6538b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:49.148982 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3559b043-776e-494f-be31-0bd196b6538b None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:49.162350 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-066c9910-3823-4ab3-93b7-a7caa6bee9cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:49.163008 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-066c9910-3823-4ab3-93b7-a7caa6bee9cc None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:49.163287 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-066c9910-3823-4ab3-93b7-a7caa6bee9cc None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:49.163588 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-066c9910-3823-4ab3-93b7-a7caa6bee9cc None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:49.163784 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-3559b043-776e-494f-be31-0bd196b6538b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:59:49.163784 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:49.163784 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:49.163784 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:49.163784 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:49.163784 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:49.163784 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:49.163784 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:49.163784 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:49.163784 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:49.163784 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:49.163784 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:49.163784 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:59:49.163784 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:59:49.163784 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:59:49.163784 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:59:49.164653 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:49.164653 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:49.164653 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:49.164653 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:49.164653 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:49.164653 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:49.164653 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:49.164653 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:49.164653 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:49.164653 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:49.164653 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:49.164653 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:49.164653 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:49.164653 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:49.164653 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:49.164653 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:49.164653 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:59:49.166148 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:59:49.166148 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:59:49.166148 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 04:59:49.166148 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 330/647] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 04:59:48 2022] GET /identity/v3/domains/Default => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:49.166148 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3559b043-776e-494f-be31-0bd196b6538b None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:49.176709 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-066c9910-3823-4ab3-93b7-a7caa6bee9cc None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:49.178169 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-066c9910-3823-4ab3-93b7-a7caa6bee9cc None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:49.189527 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-066c9910-3823-4ab3-93b7-a7caa6bee9cc None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Aug 30 04:59:49.189527 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:49.189527 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:49.189527 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:49.189527 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:49.189527 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:49.189527 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:49.189527 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:49.189527 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:49.189527 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:49.189527 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:49.189527 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:49.189527 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 04:59:49.189527 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 04:59:49.189527 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 04:59:49.189527 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 04:59:49.190474 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:49.190474 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:49.190474 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 04:59:49.190474 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:59:49.190474 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 04:59:49.190474 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:59:49.190474 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:49.190474 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:49.190474 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:49.190474 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:49.190474 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:49.190474 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:49.190474 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:49.190474 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:49.190474 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:49.190474 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:49.190474 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:49.190474 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:49.191452 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:49.191452 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:49.191452 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 04:59:49.191452 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 04:59:49.191452 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 04:59:49.191452 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 04:59:49.191452 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 04:59:49.191452 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 04:59:49.191452 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Aug 30 04:59:49.191452 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 04:59:49.191452 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 318/648] 10.222.0.77 () {56 vars in 1099 bytes} [Tue Aug 30 04:59:49 2022] GET /identity/v3/users/cinder => generated 84 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:49.192794 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-066c9910-3823-4ab3-93b7-a7caa6bee9cc None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:49.198008 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:49.279062 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3559b043-776e-494f-be31-0bd196b6538b None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:49.296517 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-066c9910-3823-4ab3-93b7-a7caa6bee9cc None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:49.311418 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1889f48-9ebe-4059-a0d5-e59de04655b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:49.312034 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1889f48-9ebe-4059-a0d5-e59de04655b4 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:49.312309 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1889f48-9ebe-4059-a0d5-e59de04655b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:49.312574 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1889f48-9ebe-4059-a0d5-e59de04655b4 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:49.320284 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1889f48-9ebe-4059-a0d5-e59de04655b4 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:49.321738 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1889f48-9ebe-4059-a0d5-e59de04655b4 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:49.324604 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:49.330541 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b946254b-a07d-4ee3-a51a-874fd21f7339 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:49.331171 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b946254b-a07d-4ee3-a51a-874fd21f7339 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:49.331452 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b946254b-a07d-4ee3-a51a-874fd21f7339 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:49.331723 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b946254b-a07d-4ee3-a51a-874fd21f7339 None admin] PATH_INFO: `/v3/domains` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:49.332361 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b946254b-a07d-4ee3-a51a-874fd21f7339 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:49.332660 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b946254b-a07d-4ee3-a51a-874fd21f7339 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:49.333754 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-d1889f48-9ebe-4059-a0d5-e59de04655b4 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 30 04:59:49.333754 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:49.333754 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:49.333754 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:49.333754 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:49.333754 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:49.333754 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:49.333754 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:49.333754 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:49.333754 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:49.333754 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:49.333754 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:49.333754 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 04:59:49.333754 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 04:59:49.333754 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 04:59:49.333754 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 04:59:49.334834 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:49.334834 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:49.334834 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 04:59:49.334834 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:59:49.334834 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 04:59:49.334834 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:59:49.334834 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:49.334834 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:49.334834 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:49.334834 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:49.334834 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:49.334834 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:49.334834 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:49.334834 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:49.334834 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:49.334834 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:49.334834 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:49.334834 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:49.336337 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:49.336337 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:49.336337 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 04:59:49.336337 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 04:59:49.336337 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 04:59:49.336337 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 04:59:49.336337 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 04:59:49.336337 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 04:59:49.336337 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 30 04:59:49.336337 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 04:59:49.336876 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b946254b-a07d-4ee3-a51a-874fd21f7339 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:49.336945 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 331/649] 10.222.0.77 () {56 vars in 1093 bytes} [Tue Aug 30 04:59:49 2022] GET /identity/v3/users/nova => generated 82 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:49.336945 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1889f48-9ebe-4059-a0d5-e59de04655b4 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:49.347696 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 319/650] 10.222.0.77 () {56 vars in 1109 bytes} [Tue Aug 30 04:59:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:49.348497 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b946254b-a07d-4ee3-a51a-874fd21f7339 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:49.420097 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:49.438264 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1889f48-9ebe-4059-a0d5-e59de04655b4 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:49.447054 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b946254b-a07d-4ee3-a51a-874fd21f7339 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:49.471947 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74178165-39fb-4267-a3f4-68a24e14ca64 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:49.472545 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74178165-39fb-4267-a3f4-68a24e14ca64 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:49.472819 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74178165-39fb-4267-a3f4-68a24e14ca64 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:49.473088 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74178165-39fb-4267-a3f4-68a24e14ca64 None admin] PATH_INFO: `/v3/domains` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:49.473712 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74178165-39fb-4267-a3f4-68a24e14ca64 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:49.474006 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74178165-39fb-4267-a3f4-68a24e14ca64 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:49.475387 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74178165-39fb-4267-a3f4-68a24e14ca64 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:49.480639 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85a9138d-a758-4f97-b5c4-70c53696de12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:49.481259 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a9138d-a758-4f97-b5c4-70c53696de12 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:49.481534 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a9138d-a758-4f97-b5c4-70c53696de12 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:49.481800 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a9138d-a758-4f97-b5c4-70c53696de12 None admin] PATH_INFO: `/v3/users` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:49.482514 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85a9138d-a758-4f97-b5c4-70c53696de12 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:49.482811 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85a9138d-a758-4f97-b5c4-70c53696de12 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:49.484247 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85a9138d-a758-4f97-b5c4-70c53696de12 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:49.487026 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 332/651] 10.222.0.77 () {56 vars in 1109 bytes} [Tue Aug 30 04:59:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:49.488173 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5044aa6f-896e-4dfa-86b5-169273479386 None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:49.488450 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5044aa6f-896e-4dfa-86b5-169273479386 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:49.488719 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5044aa6f-896e-4dfa-86b5-169273479386 None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:49.489896 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 333/652] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:59:49 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:59:49.490581 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5044aa6f-896e-4dfa-86b5-169273479386 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:49.505587 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 320/653] 10.222.0.77 () {56 vars in 1137 bytes} [Tue Aug 30 04:59:49 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:49.506393 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85a9138d-a758-4f97-b5c4-70c53696de12 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:49.555757 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:49.594072 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5044aa6f-896e-4dfa-86b5-169273479386 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:49.604964 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85a9138d-a758-4f97-b5c4-70c53696de12 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:49.628044 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4169898f-81e1-467c-bcd9-1cef5352539c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:49.628635 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4169898f-81e1-467c-bcd9-1cef5352539c None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:49.628913 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4169898f-81e1-467c-bcd9-1cef5352539c None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:49.629184 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4169898f-81e1-467c-bcd9-1cef5352539c None admin] PATH_INFO: `/v3/users` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:49.629903 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4169898f-81e1-467c-bcd9-1cef5352539c None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:49.630207 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4169898f-81e1-467c-bcd9-1cef5352539c None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:49.631665 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4169898f-81e1-467c-bcd9-1cef5352539c None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:49.638181 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac02fd2b-e478-4573-a9fb-216e9a9f5dc6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:49.638789 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac02fd2b-e478-4573-a9fb-216e9a9f5dc6 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:49.639089 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac02fd2b-e478-4573-a9fb-216e9a9f5dc6 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:49.639359 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac02fd2b-e478-4573-a9fb-216e9a9f5dc6 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:49.645590 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:49.646090 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac02fd2b-e478-4573-a9fb-216e9a9f5dc6 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:49.647570 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac02fd2b-e478-4573-a9fb-216e9a9f5dc6 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:49.649603 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 334/654] 10.222.0.77 () {56 vars in 1133 bytes} [Tue Aug 30 04:59:49 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:49.650562 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4169898f-81e1-467c-bcd9-1cef5352539c None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:49.661022 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-ac02fd2b-e478-4573-a9fb-216e9a9f5dc6 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Aug 30 04:59:49.661022 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:49.661022 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:49.661022 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:49.661022 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:49.661022 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:49.661022 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:49.661022 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:49.661022 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:49.661022 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:49.661022 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:49.661022 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:49.661022 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 04:59:49.661022 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 04:59:49.661022 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 04:59:49.661022 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 04:59:49.662017 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:49.662017 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:49.662017 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 04:59:49.662017 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:59:49.662017 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 04:59:49.662017 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:59:49.662017 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:49.662017 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:49.662017 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:49.662017 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:49.662017 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:49.662017 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:49.662017 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:49.662017 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:49.662017 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:49.662017 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:49.662017 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:49.662017 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:49.663134 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:49.663134 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:49.663134 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 04:59:49.663134 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 04:59:49.663134 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 04:59:49.663134 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 04:59:49.663134 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 04:59:49.663134 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 04:59:49.663134 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Aug 30 04:59:49.663134 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 04:59:49.663134 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 321/655] 10.222.0.77 () {56 vars in 1102 bytes} [Tue Aug 30 04:59:49 2022] GET /identity/v3/users/neutron => generated 85 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:49.663850 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be6dc999-c9e2-4da5-a57f-6e48ec5a5f8d None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:49.663850 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be6dc999-c9e2-4da5-a57f-6e48ec5a5f8d None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:49.663850 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be6dc999-c9e2-4da5-a57f-6e48ec5a5f8d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:49.681573 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-be6dc999-c9e2-4da5-a57f-6e48ec5a5f8d None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:59:49.682766 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-be6dc999-c9e2-4da5-a57f-6e48ec5a5f8d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:59:49.749369 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 322/656] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:59:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:59:49.750010 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4169898f-81e1-467c-bcd9-1cef5352539c None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:49.750208 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be6dc999-c9e2-4da5-a57f-6e48ec5a5f8d None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:49.790191 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:49.797455 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2bffbf1-1ccb-4930-be31-cd26928c440c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:49.798348 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2bffbf1-1ccb-4930-be31-cd26928c440c None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:49.798628 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2bffbf1-1ccb-4930-be31-cd26928c440c None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:49.798924 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2bffbf1-1ccb-4930-be31-cd26928c440c None admin] PATH_INFO: `/v3/projects/service` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:49.808044 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2bffbf1-1ccb-4930-be31-cd26928c440c None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:49.809922 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2bffbf1-1ccb-4930-be31-cd26928c440c None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:49.825056 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-d2bffbf1-1ccb-4930-be31-cd26928c440c None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:59:49.825056 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:49.825056 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:49.825056 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:49.825056 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:49.825056 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:49.825056 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:49.825056 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:49.825056 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:49.825056 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:49.825056 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:49.825056 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:49.825056 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 04:59:49.825056 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 04:59:49.825056 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 04:59:49.826130 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 04:59:49.826130 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:49.826130 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:49.826130 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:49.826130 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:49.826130 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:49.826130 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:49.826130 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:49.826130 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:49.826130 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:49.826130 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:49.826130 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:49.826130 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:49.826130 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:49.826130 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:49.826130 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:49.826130 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:49.827055 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 04:59:49.827055 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 04:59:49.827055 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 04:59:49.827055 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 04:59:49.827055 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 04:59:49.827055 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 04:59:49.827055 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:59:49.827055 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 04:59:49.827055 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 335/657] 10.222.0.77 () {56 vars in 1111 bytes} [Tue Aug 30 04:59:49 2022] GET /identity/v3/projects/service => generated 88 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:49.827547 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2bffbf1-1ccb-4930-be31-cd26928c440c None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:49.857858 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be6dc999-c9e2-4da5-a57f-6e48ec5a5f8d None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:49.896434 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:49.898397 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6302f53-85ec-4cb0-9222-b2848ecb5ec4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:49.899214 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6302f53-85ec-4cb0-9222-b2848ecb5ec4 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:49.899652 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6302f53-85ec-4cb0-9222-b2848ecb5ec4 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:49.900112 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6302f53-85ec-4cb0-9222-b2848ecb5ec4 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:49.940696 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2bffbf1-1ccb-4930-be31-cd26928c440c None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:49.971017 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6302f53-85ec-4cb0-9222-b2848ecb5ec4 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:49.972456 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6302f53-85ec-4cb0-9222-b2848ecb5ec4 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:49.975446 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b94094fa-0d8f-4574-b6b5-8dd8a9c0254d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:49.975747 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b94094fa-0d8f-4574-b6b5-8dd8a9c0254d None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:49.976491 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b94094fa-0d8f-4574-b6b5-8dd8a9c0254d None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:49.976761 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b94094fa-0d8f-4574-b6b5-8dd8a9c0254d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:50.013528 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:50.044668 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b94094fa-0d8f-4574-b6b5-8dd8a9c0254d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:50.046093 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b94094fa-0d8f-4574-b6b5-8dd8a9c0254d None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:50.074877 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 323/658] 10.222.0.77 () {58 vars in 1282 bytes} [Tue Aug 30 04:59:49 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 325 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 04:59:50.075646 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6302f53-85ec-4cb0-9222-b2848ecb5ec4 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:50.103526 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:50.145591 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 336/659] 10.222.0.77 () {58 vars in 1282 bytes} [Tue Aug 30 04:59:49 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 319 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 04:59:50.146409 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b94094fa-0d8f-4574-b6b5-8dd8a9c0254d None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:50.175337 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6302f53-85ec-4cb0-9222-b2848ecb5ec4 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:50.208431 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d73e936-4560-4d69-90ef-c2dd8b6c314c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:50.209017 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d73e936-4560-4d69-90ef-c2dd8b6c314c None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:50.209288 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d73e936-4560-4d69-90ef-c2dd8b6c314c None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:50.209550 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d73e936-4560-4d69-90ef-c2dd8b6c314c None admin] PATH_INFO: `/v3/users` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:50.210761 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d73e936-4560-4d69-90ef-c2dd8b6c314c None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:50.211102 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d73e936-4560-4d69-90ef-c2dd8b6c314c None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:50.212528 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d73e936-4560-4d69-90ef-c2dd8b6c314c None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:50.220205 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:50.229684 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 324/660] 10.222.0.77 () {56 vars in 1139 bytes} [Tue Aug 30 04:59:50 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:50.230662 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d73e936-4560-4d69-90ef-c2dd8b6c314c None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:50.246238 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b94094fa-0d8f-4574-b6b5-8dd8a9c0254d None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:50.279575 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49e4f9ea-2430-4efe-80ac-9db475011c0c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:50.280176 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49e4f9ea-2430-4efe-80ac-9db475011c0c None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:50.280459 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49e4f9ea-2430-4efe-80ac-9db475011c0c None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:50.280727 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49e4f9ea-2430-4efe-80ac-9db475011c0c None admin] PATH_INFO: `/v3/projects` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:50.281405 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49e4f9ea-2430-4efe-80ac-9db475011c0c None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:50.281705 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49e4f9ea-2430-4efe-80ac-9db475011c0c None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:50.283166 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49e4f9ea-2430-4efe-80ac-9db475011c0c None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:50.294317 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 337/661] 10.222.0.77 () {56 vars in 1148 bytes} [Tue Aug 30 04:59:50 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:50.295125 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49e4f9ea-2430-4efe-80ac-9db475011c0c None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:50.316025 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:50.332446 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d73e936-4560-4d69-90ef-c2dd8b6c314c None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:50.364887 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ab6f685-bc35-47bc-bc51-999cd7a0cef5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:50.365504 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ab6f685-bc35-47bc-bc51-999cd7a0cef5 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:50.365785 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ab6f685-bc35-47bc-bc51-999cd7a0cef5 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:50.366059 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ab6f685-bc35-47bc-bc51-999cd7a0cef5 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:50.376983 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ab6f685-bc35-47bc-bc51-999cd7a0cef5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:59:50.381751 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-2ab6f685-bc35-47bc-bc51-999cd7a0cef5 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 30 04:59:50.381751 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:50.381751 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:50.381751 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:50.381751 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:50.381751 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:50.381751 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:50.381751 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:50.381751 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:50.381751 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:50.381751 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:50.381751 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:50.381751 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:59:50.381751 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:59:50.381751 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:59:50.381751 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:59:50.383343 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:59:50.383343 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:59:50.383343 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:59:50.383343 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:59:50.383343 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:50.383343 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:50.383343 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:50.383343 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:50.383343 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:50.383343 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:50.383343 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:50.383343 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:50.383343 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:50.383343 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:50.383343 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:50.383343 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:50.387950 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:50.387950 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:50.387950 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:50.387950 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:50.387950 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:59:50.387950 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:59:50.387950 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:59:50.387950 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:59:50.387950 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:59:50.387950 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:59:50.387950 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 30 04:59:50.387950 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 04:59:50.387950 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 325/662] 10.222.0.77 () {56 vars in 1102 bytes} [Tue Aug 30 04:59:50 2022] GET /identity/v3/roles/service => generated 85 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:50.387950 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ab6f685-bc35-47bc-bc51-999cd7a0cef5 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:50.392128 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49e4f9ea-2430-4efe-80ac-9db475011c0c None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:50.424544 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32e55f27-e587-4e4c-a1b0-26db67b01834 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:50.425151 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e55f27-e587-4e4c-a1b0-26db67b01834 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:50.425425 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e55f27-e587-4e4c-a1b0-26db67b01834 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:50.425690 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e55f27-e587-4e4c-a1b0-26db67b01834 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:50.436325 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32e55f27-e587-4e4c-a1b0-26db67b01834 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:50.437754 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32e55f27-e587-4e4c-a1b0-26db67b01834 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:50.439953 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:50.457023 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-32e55f27-e587-4e4c-a1b0-26db67b01834 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:59:50.457023 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:50.457023 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:50.457023 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:50.457023 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:50.457023 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:50.457023 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:50.457023 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:50.457023 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:50.457023 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:50.457023 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:50.457023 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:50.457023 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:59:50.457023 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:59:50.457023 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:59:50.457023 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:59:50.458265 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:50.458265 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:50.458265 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:50.458265 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:50.458265 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:50.458265 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:50.458265 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:50.458265 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:50.458265 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:50.458265 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:50.458265 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:50.458265 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:50.458265 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:50.458265 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:50.458265 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:50.458265 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:50.458265 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:59:50.464099 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:59:50.464099 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:59:50.464099 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 04:59:50.464099 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 338/663] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 04:59:50 2022] GET /identity/v3/domains/Default => generated 87 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:50.464099 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32e55f27-e587-4e4c-a1b0-26db67b01834 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:50.490976 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ab6f685-bc35-47bc-bc51-999cd7a0cef5 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:50.525974 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f748646d-bc2e-4fd4-8ab3-02fe33978904 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:50.526595 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f748646d-bc2e-4fd4-8ab3-02fe33978904 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:50.526903 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f748646d-bc2e-4fd4-8ab3-02fe33978904 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:50.527173 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f748646d-bc2e-4fd4-8ab3-02fe33978904 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:50.528685 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:50.537185 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f748646d-bc2e-4fd4-8ab3-02fe33978904 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:50.538628 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f748646d-bc2e-4fd4-8ab3-02fe33978904 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:50.552866 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-f748646d-bc2e-4fd4-8ab3-02fe33978904 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:59:50.552866 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:50.552866 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:50.552866 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:50.552866 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:50.552866 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:50.552866 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:50.552866 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:50.552866 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:50.552866 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:50.552866 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:50.552866 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:50.552866 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:59:50.552866 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:59:50.552866 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:59:50.552866 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:59:50.553894 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:50.553894 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:50.553894 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:50.553894 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:50.553894 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:50.553894 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:50.553894 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:50.553894 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:50.553894 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:50.553894 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:50.553894 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:50.553894 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:50.553894 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:50.553894 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:50.553894 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:50.553894 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:50.553894 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:59:50.555338 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:59:50.555338 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:59:50.555338 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 04:59:50.555338 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 326/664] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 04:59:50 2022] GET /identity/v3/domains/Default => generated 87 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:50.555338 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f748646d-bc2e-4fd4-8ab3-02fe33978904 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:50.562584 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32e55f27-e587-4e4c-a1b0-26db67b01834 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:50.595398 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7e44e75-9645-4200-8307-da24d54fb384 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:50.595916 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7e44e75-9645-4200-8307-da24d54fb384 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:50.596184 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7e44e75-9645-4200-8307-da24d54fb384 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:50.596445 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7e44e75-9645-4200-8307-da24d54fb384 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:50.651389 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f748646d-bc2e-4fd4-8ab3-02fe33978904 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:50.659890 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:50.661670 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7e44e75-9645-4200-8307-da24d54fb384 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:50.663153 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7e44e75-9645-4200-8307-da24d54fb384 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:50.686089 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f107555-85a4-468f-a26f-41b563dfc0a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:50.686683 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f107555-85a4-468f-a26f-41b563dfc0a5 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:50.686995 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f107555-85a4-468f-a26f-41b563dfc0a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:50.687266 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f107555-85a4-468f-a26f-41b563dfc0a5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:50.687978 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f107555-85a4-468f-a26f-41b563dfc0a5 None admin] RBAC: Adding query filter params (role.id=821647e8b40542499667e29e1b301cb2, scope.project.id=e9919141e3b947ab8d91b55bba4a7de6, user.id=e0a002ac0bf240a891ac7304178d3f0e) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:50.688283 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f107555-85a4-468f-a26f-41b563dfc0a5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:50.689704 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f107555-85a4-468f-a26f-41b563dfc0a5 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:50.695269 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 327/665] 10.222.0.77 () {56 vars in 1374 bytes} [Tue Aug 30 04:59:50 2022] GET /identity/v3/role_assignments?user.id=e0a002ac0bf240a891ac7304178d3f0e&scope.project.id=e9919141e3b947ab8d91b55bba4a7de6&role.id=821647e8b40542499667e29e1b301cb2 => generated 608 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:50.696030 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f107555-85a4-468f-a26f-41b563dfc0a5 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:50.736643 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:50.770922 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 339/666] 10.222.0.77 () {58 vars in 1282 bytes} [Tue Aug 30 04:59:50 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 04:59:50.772690 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7e44e75-9645-4200-8307-da24d54fb384 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:50.825314 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f107555-85a4-468f-a26f-41b563dfc0a5 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:50.860531 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5505574c-4945-4f82-bc5b-52914fdcd271 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:50.861061 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5505574c-4945-4f82-bc5b-52914fdcd271 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:50.861338 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5505574c-4945-4f82-bc5b-52914fdcd271 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:50.861609 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5505574c-4945-4f82-bc5b-52914fdcd271 None admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:50.862268 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5505574c-4945-4f82-bc5b-52914fdcd271 None admin] RBAC: Adding query filter params (name=service) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:50.862567 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5505574c-4945-4f82-bc5b-52914fdcd271 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:50.863974 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5505574c-4945-4f82-bc5b-52914fdcd271 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:50.873394 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 328/667] 10.222.0.77 () {56 vars in 1103 bytes} [Tue Aug 30 04:59:50 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:50.874221 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5505574c-4945-4f82-bc5b-52914fdcd271 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:50.889301 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7e44e75-9645-4200-8307-da24d54fb384 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:50.917462 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:50.929909 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-001f47a4-c57e-4433-8c81-3ff41be25e4e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:50.930994 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001f47a4-c57e-4433-8c81-3ff41be25e4e None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:50.931492 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001f47a4-c57e-4433-8c81-3ff41be25e4e None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:50.931918 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001f47a4-c57e-4433-8c81-3ff41be25e4e None admin] PATH_INFO: `/v3/domains` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:50.932710 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-001f47a4-c57e-4433-8c81-3ff41be25e4e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:50.933247 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-001f47a4-c57e-4433-8c81-3ff41be25e4e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:50.934920 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-001f47a4-c57e-4433-8c81-3ff41be25e4e None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:50.948950 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 340/668] 10.222.0.77 () {56 vars in 1109 bytes} [Tue Aug 30 04:59:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:50.949978 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-001f47a4-c57e-4433-8c81-3ff41be25e4e None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:50.984762 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:50.993298 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5505574c-4945-4f82-bc5b-52914fdcd271 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:51.027920 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98c43330-f28e-4cc8-90bb-223b0da92ad6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:51.028530 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98c43330-f28e-4cc8-90bb-223b0da92ad6 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:51.028808 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98c43330-f28e-4cc8-90bb-223b0da92ad6 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:51.029080 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98c43330-f28e-4cc8-90bb-223b0da92ad6 None admin] PATH_INFO: `/v3/domains` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:51.029713 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98c43330-f28e-4cc8-90bb-223b0da92ad6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:51.030015 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98c43330-f28e-4cc8-90bb-223b0da92ad6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:51.031411 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98c43330-f28e-4cc8-90bb-223b0da92ad6 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:51.044235 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 329/669] 10.222.0.77 () {56 vars in 1109 bytes} [Tue Aug 30 04:59:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:51.045061 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98c43330-f28e-4cc8-90bb-223b0da92ad6 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:51.064324 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-001f47a4-c57e-4433-8c81-3ff41be25e4e None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:51.098777 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45112b18-6bec-490c-a0f9-1b538251d379 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:51.099459 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45112b18-6bec-490c-a0f9-1b538251d379 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:51.099736 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45112b18-6bec-490c-a0f9-1b538251d379 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:51.100002 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45112b18-6bec-490c-a0f9-1b538251d379 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:51.112239 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45112b18-6bec-490c-a0f9-1b538251d379 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:51.113624 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45112b18-6bec-490c-a0f9-1b538251d379 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:51.127499 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-45112b18-6bec-490c-a0f9-1b538251d379 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:59:51.127499 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:51.127499 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:51.127499 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:51.127499 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:51.127499 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:51.127499 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:51.127499 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:51.127499 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:51.127499 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:51.127499 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:51.127499 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:51.127499 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:59:51.127499 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:59:51.127499 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:59:51.127499 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:59:51.128638 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:51.128638 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:51.128638 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:51.128638 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:51.128638 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:51.128638 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:51.128638 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:51.128638 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:51.128638 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:51.128638 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:51.128638 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:51.128638 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:51.128638 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:51.128638 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:51.128638 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:51.128638 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:51.128638 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:59:51.129547 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:59:51.129547 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:59:51.129547 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 04:59:51.129547 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 341/670] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 04:59:50 2022] GET /identity/v3/domains/Default => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:51.129547 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45112b18-6bec-490c-a0f9-1b538251d379 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:51.149637 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:51.158548 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98c43330-f28e-4cc8-90bb-223b0da92ad6 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:51.196614 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8502a3a-46dd-4844-b4e7-792ab24a8e6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:51.197477 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8502a3a-46dd-4844-b4e7-792ab24a8e6f None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:51.198011 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8502a3a-46dd-4844-b4e7-792ab24a8e6f None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:51.198446 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8502a3a-46dd-4844-b4e7-792ab24a8e6f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:51.209890 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8502a3a-46dd-4844-b4e7-792ab24a8e6f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:51.211845 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8502a3a-46dd-4844-b4e7-792ab24a8e6f None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:51.221380 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:51.228532 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-b8502a3a-46dd-4844-b4e7-792ab24a8e6f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:59:51.228532 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:51.228532 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:51.228532 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:51.228532 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:51.228532 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:51.228532 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:51.228532 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:51.228532 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:51.228532 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:51.228532 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:51.228532 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:51.228532 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:59:51.228532 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:59:51.228532 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:59:51.228532 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:59:51.229521 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:51.229521 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:51.229521 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:51.229521 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:51.229521 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:51.229521 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:51.229521 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:51.229521 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:51.229521 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:51.229521 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:51.229521 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:51.229521 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:51.229521 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:51.229521 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:51.229521 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:51.229521 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:51.229521 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:59:51.230495 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:59:51.230495 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:59:51.230495 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 04:59:51.230495 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 330/671] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 04:59:51 2022] GET /identity/v3/domains/Default => generated 87 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:51.230495 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8502a3a-46dd-4844-b4e7-792ab24a8e6f None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:51.247547 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45112b18-6bec-490c-a0f9-1b538251d379 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:51.283665 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a939ca1-c959-4e79-8ccb-24baeb1060ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:51.284341 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a939ca1-c959-4e79-8ccb-24baeb1060ea None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:51.284705 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a939ca1-c959-4e79-8ccb-24baeb1060ea None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:51.285046 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a939ca1-c959-4e79-8ccb-24baeb1060ea None admin] PATH_INFO: `/v3/projects/service` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:51.293372 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a939ca1-c959-4e79-8ccb-24baeb1060ea None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:51.295264 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a939ca1-c959-4e79-8ccb-24baeb1060ea None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:51.309265 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-6a939ca1-c959-4e79-8ccb-24baeb1060ea None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:59:51.309265 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:51.309265 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:51.309265 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:51.309265 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:51.309265 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:51.309265 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:51.309265 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:51.309265 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:51.309265 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:51.309265 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:51.309265 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:51.309265 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 04:59:51.309265 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 04:59:51.309265 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 04:59:51.310187 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 04:59:51.310187 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:51.310187 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:51.310187 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:51.310187 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:51.310187 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:51.310187 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:51.310187 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:51.310187 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:51.310187 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:51.310187 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:51.310187 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:51.310187 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:51.310187 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:51.310187 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:51.310187 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:51.310187 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:51.311127 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 04:59:51.311127 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 04:59:51.311127 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 04:59:51.311127 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 04:59:51.311127 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 04:59:51.311127 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 04:59:51.311127 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:59:51.311127 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 04:59:51.312251 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 342/672] 10.222.0.77 () {56 vars in 1111 bytes} [Tue Aug 30 04:59:51 2022] GET /identity/v3/projects/service => generated 88 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:51.313136 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a939ca1-c959-4e79-8ccb-24baeb1060ea None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:51.340082 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8502a3a-46dd-4844-b4e7-792ab24a8e6f None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:51.374925 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2bc0976-6a0e-458f-8f6e-b35be959096c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:51.375638 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2bc0976-6a0e-458f-8f6e-b35be959096c None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:51.375916 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2bc0976-6a0e-458f-8f6e-b35be959096c None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:51.376185 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2bc0976-6a0e-458f-8f6e-b35be959096c None admin] PATH_INFO: `/v3/projects/service` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:51.383413 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2bc0976-6a0e-458f-8f6e-b35be959096c None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:51.384923 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2bc0976-6a0e-458f-8f6e-b35be959096c None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:51.396257 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-d2bc0976-6a0e-458f-8f6e-b35be959096c None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:59:51.396257 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:51.396257 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:51.396257 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:51.396257 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:51.396257 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:51.396257 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:51.396257 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:51.396257 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:51.396257 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:51.396257 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:51.396257 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:51.396257 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 04:59:51.396257 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 04:59:51.396257 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 04:59:51.397348 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 04:59:51.397348 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:51.397348 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:51.397348 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:51.397348 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:51.397348 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:51.397348 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:51.397348 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:51.397348 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:51.397348 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:51.397348 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:51.397348 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:51.397348 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:51.397348 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:51.397348 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:51.397348 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:51.397348 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:51.398883 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 04:59:51.398883 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 04:59:51.398883 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 04:59:51.398883 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 04:59:51.398883 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 04:59:51.398883 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 04:59:51.398883 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:59:51.398883 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 04:59:51.398883 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 331/673] 10.222.0.77 () {56 vars in 1111 bytes} [Tue Aug 30 04:59:51 2022] GET /identity/v3/projects/service => generated 88 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:51.399385 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2bc0976-6a0e-458f-8f6e-b35be959096c None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:51.411619 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:51.425826 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a939ca1-c959-4e79-8ccb-24baeb1060ea None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:51.468155 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-840ed81b-b5a7-4c9f-b1c4-355fa562c55b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:51.468955 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-840ed81b-b5a7-4c9f-b1c4-355fa562c55b None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:51.469452 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-840ed81b-b5a7-4c9f-b1c4-355fa562c55b None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:51.469562 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-840ed81b-b5a7-4c9f-b1c4-355fa562c55b None admin] PATH_INFO: `/v3/domains` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:51.470117 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-840ed81b-b5a7-4c9f-b1c4-355fa562c55b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:51.470416 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-840ed81b-b5a7-4c9f-b1c4-355fa562c55b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:51.471795 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-840ed81b-b5a7-4c9f-b1c4-355fa562c55b None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:51.483900 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 343/674] 10.222.0.77 () {56 vars in 1109 bytes} [Tue Aug 30 04:59:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:51.484529 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-840ed81b-b5a7-4c9f-b1c4-355fa562c55b None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:51.488683 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:51.516245 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2bc0976-6a0e-458f-8f6e-b35be959096c None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:51.551621 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af7753f3-1da2-4d13-9c16-72909bdbc99c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:51.552336 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af7753f3-1da2-4d13-9c16-72909bdbc99c None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:51.552709 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af7753f3-1da2-4d13-9c16-72909bdbc99c None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:51.553075 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af7753f3-1da2-4d13-9c16-72909bdbc99c None admin] PATH_INFO: `/v3/domains` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:51.553818 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af7753f3-1da2-4d13-9c16-72909bdbc99c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:51.554113 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af7753f3-1da2-4d13-9c16-72909bdbc99c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:51.555505 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af7753f3-1da2-4d13-9c16-72909bdbc99c None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:51.569087 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 332/675] 10.222.0.77 () {56 vars in 1109 bytes} [Tue Aug 30 04:59:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:51.570358 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af7753f3-1da2-4d13-9c16-72909bdbc99c None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:51.594307 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-840ed81b-b5a7-4c9f-b1c4-355fa562c55b None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:51.629996 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c338aab-ffff-4d1d-98af-186a848301ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:51.630938 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c338aab-ffff-4d1d-98af-186a848301ad None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:51.631357 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c338aab-ffff-4d1d-98af-186a848301ad None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:51.631845 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c338aab-ffff-4d1d-98af-186a848301ad None admin] PATH_INFO: `/v3/projects` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:51.633170 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c338aab-ffff-4d1d-98af-186a848301ad None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:51.633608 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c338aab-ffff-4d1d-98af-186a848301ad None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:51.635385 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:51.636253 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c338aab-ffff-4d1d-98af-186a848301ad None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:51.654710 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 344/676] 10.222.0.77 () {56 vars in 1148 bytes} [Tue Aug 30 04:59:51 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:51.655912 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c338aab-ffff-4d1d-98af-186a848301ad None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:51.671249 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af7753f3-1da2-4d13-9c16-72909bdbc99c None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:51.706462 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdb8f486-0c9d-44fb-88cb-42cb62d2ad19 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:51.707374 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdb8f486-0c9d-44fb-88cb-42cb62d2ad19 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:51.707662 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdb8f486-0c9d-44fb-88cb-42cb62d2ad19 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:51.707923 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdb8f486-0c9d-44fb-88cb-42cb62d2ad19 None admin] PATH_INFO: `/v3/projects` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:51.708605 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdb8f486-0c9d-44fb-88cb-42cb62d2ad19 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:51.708900 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdb8f486-0c9d-44fb-88cb-42cb62d2ad19 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:51.710344 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdb8f486-0c9d-44fb-88cb-42cb62d2ad19 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:51.721795 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 333/677] 10.222.0.77 () {56 vars in 1148 bytes} [Tue Aug 30 04:59:51 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:51.722585 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdb8f486-0c9d-44fb-88cb-42cb62d2ad19 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:51.723554 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:51.762767 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c338aab-ffff-4d1d-98af-186a848301ad None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:51.795374 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2af311b7-7cc8-4326-926a-f458c089027b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:51.795992 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af311b7-7cc8-4326-926a-f458c089027b None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:51.796273 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af311b7-7cc8-4326-926a-f458c089027b None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:51.796542 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af311b7-7cc8-4326-926a-f458c089027b None admin] PATH_INFO: `/v3/projects/service` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:51.803058 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2af311b7-7cc8-4326-926a-f458c089027b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:51.805055 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2af311b7-7cc8-4326-926a-f458c089027b None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:51.816287 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-2af311b7-7cc8-4326-926a-f458c089027b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:59:51.816287 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:51.816287 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:51.816287 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:51.816287 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:51.816287 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:51.816287 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:51.816287 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:51.816287 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:51.816287 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:51.816287 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:51.816287 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:51.816287 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 04:59:51.816287 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 04:59:51.816287 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 04:59:51.817339 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 04:59:51.817339 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:51.817339 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:51.817339 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:51.817339 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:51.817339 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:51.817339 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:51.817339 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:51.817339 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:51.817339 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:51.817339 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:51.817339 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:51.817339 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:51.817339 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:51.817339 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:51.817339 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:51.817339 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:51.818259 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 04:59:51.818259 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 04:59:51.818259 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 04:59:51.818259 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 04:59:51.818259 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 04:59:51.818259 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 04:59:51.818259 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:59:51.818259 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 04:59:51.818259 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 345/678] 10.222.0.77 () {56 vars in 1111 bytes} [Tue Aug 30 04:59:51 2022] GET /identity/v3/projects/service => generated 88 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:51.818259 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2af311b7-7cc8-4326-926a-f458c089027b None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:51.838131 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdb8f486-0c9d-44fb-88cb-42cb62d2ad19 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:51.865834 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:51.876975 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87d2b703-5acc-4719-bd46-378dc7e72569 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:51.877593 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d2b703-5acc-4719-bd46-378dc7e72569 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:51.877874 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d2b703-5acc-4719-bd46-378dc7e72569 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:51.878145 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d2b703-5acc-4719-bd46-378dc7e72569 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:51.886173 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87d2b703-5acc-4719-bd46-378dc7e72569 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:51.887586 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87d2b703-5acc-4719-bd46-378dc7e72569 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:51.898848 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-87d2b703-5acc-4719-bd46-378dc7e72569 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Aug 30 04:59:51.898848 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:51.898848 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:51.898848 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:51.898848 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:51.898848 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:51.898848 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:51.898848 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:51.898848 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:51.898848 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:51.898848 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:51.898848 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:51.898848 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 04:59:51.898848 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 04:59:51.898848 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 04:59:51.898848 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 04:59:51.899986 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:51.899986 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:51.899986 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 04:59:51.899986 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:59:51.899986 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 04:59:51.899986 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:59:51.899986 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:51.899986 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:51.899986 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:51.899986 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:51.899986 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:51.899986 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:51.899986 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:51.899986 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:51.899986 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:51.899986 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:51.899986 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:51.899986 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:51.901206 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:51.901206 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:51.901206 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 04:59:51.901206 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 04:59:51.901206 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 04:59:51.901206 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 04:59:51.901206 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 04:59:51.901206 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 04:59:51.901206 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Aug 30 04:59:51.901206 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 04:59:51.901206 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 334/679] 10.222.0.77 () {56 vars in 1096 bytes} [Tue Aug 30 04:59:51 2022] GET /identity/v3/users/swift => generated 83 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:51.901206 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87d2b703-5acc-4719-bd46-378dc7e72569 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:51.930523 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2af311b7-7cc8-4326-926a-f458c089027b None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:51.954834 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:51.967453 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0c555ff-d34b-4ec8-aa2c-3a1825915bc9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:51.968052 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0c555ff-d34b-4ec8-aa2c-3a1825915bc9 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:51.968327 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0c555ff-d34b-4ec8-aa2c-3a1825915bc9 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:51.968593 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0c555ff-d34b-4ec8-aa2c-3a1825915bc9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:51.969305 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0c555ff-d34b-4ec8-aa2c-3a1825915bc9 None admin] RBAC: Adding query filter params (role.id=821647e8b40542499667e29e1b301cb2, scope.project.id=e9919141e3b947ab8d91b55bba4a7de6, user.id=019a79b2affe478b8a30d1c538d0415c) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:51.969609 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0c555ff-d34b-4ec8-aa2c-3a1825915bc9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:51.971046 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0c555ff-d34b-4ec8-aa2c-3a1825915bc9 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:51.978939 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 346/680] 10.222.0.77 () {56 vars in 1374 bytes} [Tue Aug 30 04:59:51 2022] GET /identity/v3/role_assignments?user.id=019a79b2affe478b8a30d1c538d0415c&scope.project.id=e9919141e3b947ab8d91b55bba4a7de6&role.id=821647e8b40542499667e29e1b301cb2 => generated 608 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:51.978939 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0c555ff-d34b-4ec8-aa2c-3a1825915bc9 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:52.027633 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87d2b703-5acc-4719-bd46-378dc7e72569 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:52.069432 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a57531f-d896-4b41-b3ca-9472c0c96ac7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:52.070936 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a57531f-d896-4b41-b3ca-9472c0c96ac7 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:52.071361 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a57531f-d896-4b41-b3ca-9472c0c96ac7 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:52.071862 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a57531f-d896-4b41-b3ca-9472c0c96ac7 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:52.072811 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a57531f-d896-4b41-b3ca-9472c0c96ac7 None admin] RBAC: Adding query filter params (role.id=821647e8b40542499667e29e1b301cb2, scope.project.id=e9919141e3b947ab8d91b55bba4a7de6, user.id=886eb5e00bbf4ec292eb962b665b37d0) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:52.073341 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a57531f-d896-4b41-b3ca-9472c0c96ac7 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:52.075078 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a57531f-d896-4b41-b3ca-9472c0c96ac7 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:52.082165 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 335/681] 10.222.0.77 () {56 vars in 1374 bytes} [Tue Aug 30 04:59:51 2022] GET /identity/v3/role_assignments?user.id=886eb5e00bbf4ec292eb962b665b37d0&scope.project.id=e9919141e3b947ab8d91b55bba4a7de6&role.id=821647e8b40542499667e29e1b301cb2 => generated 608 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:52.083434 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a57531f-d896-4b41-b3ca-9472c0c96ac7 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:52.139576 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0c555ff-d34b-4ec8-aa2c-3a1825915bc9 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:52.172876 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:52.183939 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da766cea-b090-420e-a1ac-ee8e321aedb2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:52.184285 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da766cea-b090-420e-a1ac-ee8e321aedb2 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:52.184557 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da766cea-b090-420e-a1ac-ee8e321aedb2 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:52.184829 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da766cea-b090-420e-a1ac-ee8e321aedb2 None admin] PATH_INFO: `/v3/projects` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:52.185535 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da766cea-b090-420e-a1ac-ee8e321aedb2 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:52.185898 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da766cea-b090-420e-a1ac-ee8e321aedb2 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:52.187366 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da766cea-b090-420e-a1ac-ee8e321aedb2 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:52.203241 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 347/682] 10.222.0.77 () {56 vars in 1148 bytes} [Tue Aug 30 04:59:51 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:52.209159 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da766cea-b090-420e-a1ac-ee8e321aedb2 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:52.246165 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a57531f-d896-4b41-b3ca-9472c0c96ac7 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:52.274382 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:52.292709 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29d8309e-ca7a-4633-afd8-77a67154500f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:52.293588 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29d8309e-ca7a-4633-afd8-77a67154500f None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:52.294167 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29d8309e-ca7a-4633-afd8-77a67154500f None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:52.294609 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29d8309e-ca7a-4633-afd8-77a67154500f None admin] PATH_INFO: `/v3/users` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:52.295362 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29d8309e-ca7a-4633-afd8-77a67154500f None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:52.295672 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29d8309e-ca7a-4633-afd8-77a67154500f None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:52.297901 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29d8309e-ca7a-4633-afd8-77a67154500f None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:52.325635 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 336/683] 10.222.0.77 () {56 vars in 1135 bytes} [Tue Aug 30 04:59:52 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 241 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:52.330639 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29d8309e-ca7a-4633-afd8-77a67154500f None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:52.344012 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da766cea-b090-420e-a1ac-ee8e321aedb2 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:52.385270 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1b4e294-223b-44c8-9379-2d5ef7a57e02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:52.386292 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1b4e294-223b-44c8-9379-2d5ef7a57e02 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:52.386954 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1b4e294-223b-44c8-9379-2d5ef7a57e02 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:52.387483 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1b4e294-223b-44c8-9379-2d5ef7a57e02 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:52.388475 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1b4e294-223b-44c8-9379-2d5ef7a57e02 None admin] RBAC: Adding query filter params (role.id=821647e8b40542499667e29e1b301cb2, scope.project.id=e9919141e3b947ab8d91b55bba4a7de6, user.id=c33c45f4770d4151858e1d897d09af69) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:52.388997 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1b4e294-223b-44c8-9379-2d5ef7a57e02 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:52.390843 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1b4e294-223b-44c8-9379-2d5ef7a57e02 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:52.398188 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 348/684] 10.222.0.77 () {56 vars in 1374 bytes} [Tue Aug 30 04:59:52 2022] GET /identity/v3/role_assignments?user.id=c33c45f4770d4151858e1d897d09af69&scope.project.id=e9919141e3b947ab8d91b55bba4a7de6&role.id=821647e8b40542499667e29e1b301cb2 => generated 608 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:52.444099 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29d8309e-ca7a-4633-afd8-77a67154500f None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:52.455376 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:52.484482 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3b1dc2c-483f-478b-9379-af317a79e0db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:52.485103 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3b1dc2c-483f-478b-9379-af317a79e0db None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:52.485381 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3b1dc2c-483f-478b-9379-af317a79e0db None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:52.485651 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3b1dc2c-483f-478b-9379-af317a79e0db None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:52.561491 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3b1dc2c-483f-478b-9379-af317a79e0db None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:52.562996 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3b1dc2c-483f-478b-9379-af317a79e0db None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:52.671464 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 337/685] 10.222.0.77 () {58 vars in 1282 bytes} [Tue Aug 30 04:59:52 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 341 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Aug 30 04:59:52.676976 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1b4e294-223b-44c8-9379-2d5ef7a57e02 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:52.730387 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8686be5d-17a1-415f-8b16-49753446526f None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:52.730722 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8686be5d-17a1-415f-8b16-49753446526f None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:52.731022 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8686be5d-17a1-415f-8b16-49753446526f None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:52.732567 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 338/686] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:59:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:59:52.740233 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e96474f6-d4bb-43e3-bb7c-cea34355d270 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:52.740615 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e96474f6-d4bb-43e3-bb7c-cea34355d270 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:52.740807 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e96474f6-d4bb-43e3-bb7c-cea34355d270 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:52.766054 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-e96474f6-d4bb-43e3-bb7c-cea34355d270 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:59:52.767290 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-e96474f6-d4bb-43e3-bb7c-cea34355d270 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:59:52.771122 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:52.800001 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1b4e294-223b-44c8-9379-2d5ef7a57e02 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:52.813854 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:52.843239 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1687189-2733-4e1d-9301-021dc45a148f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:52.843881 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1687189-2733-4e1d-9301-021dc45a148f None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:52.844154 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1687189-2733-4e1d-9301-021dc45a148f None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:52.844418 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1687189-2733-4e1d-9301-021dc45a148f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:52.870421 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1687189-2733-4e1d-9301-021dc45a148f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:52.871279 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 339/687] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:59:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:59:52.871909 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1687189-2733-4e1d-9301-021dc45a148f None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:52.900544 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-a1687189-2733-4e1d-9301-021dc45a148f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:59:52.900544 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:52.900544 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:52.900544 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:52.900544 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:52.900544 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:52.900544 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:52.900544 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:52.900544 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:52.900544 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:52.900544 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:52.900544 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:52.900544 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:59:52.900544 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:59:52.900544 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:59:52.900544 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:59:52.901534 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:52.901534 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:52.901534 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:52.901534 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:52.901534 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:52.901534 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:52.901534 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:52.901534 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:52.901534 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:52.901534 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:52.901534 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:52.901534 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:52.901534 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:52.901534 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:52.901534 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:52.901534 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:52.901534 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:59:52.902400 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:59:52.902400 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:59:52.902400 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 04:59:52.904635 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 349/688] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 04:59:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 228 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:52.910511 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e96474f6-d4bb-43e3-bb7c-cea34355d270 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:53.045796 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e96474f6-d4bb-43e3-bb7c-cea34355d270 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:53.093012 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bf50ea5-7659-4ba5-adf6-791ef4a014e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:53.093620 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf50ea5-7659-4ba5-adf6-791ef4a014e2 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:53.093896 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf50ea5-7659-4ba5-adf6-791ef4a014e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:53.094163 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf50ea5-7659-4ba5-adf6-791ef4a014e2 None admin] PATH_INFO: `/v3/domains` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:53.094793 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bf50ea5-7659-4ba5-adf6-791ef4a014e2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:53.095119 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bf50ea5-7659-4ba5-adf6-791ef4a014e2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:53.103932 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bf50ea5-7659-4ba5-adf6-791ef4a014e2 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:53.110936 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:53.120027 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 340/689] 10.222.0.77 () {56 vars in 1109 bytes} [Tue Aug 30 04:59:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:53.127753 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1687189-2733-4e1d-9301-021dc45a148f None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:53.167008 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bf50ea5-7659-4ba5-adf6-791ef4a014e2 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:53.251477 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1687189-2733-4e1d-9301-021dc45a148f None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:53.289523 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52199433-5919-4c03-8b52-3322c1ebac0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:53.290547 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52199433-5919-4c03-8b52-3322c1ebac0d None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:53.291185 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52199433-5919-4c03-8b52-3322c1ebac0d None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:53.291673 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52199433-5919-4c03-8b52-3322c1ebac0d None admin] PATH_INFO: `/v3/projects/service` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:53.299099 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52199433-5919-4c03-8b52-3322c1ebac0d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:53.301102 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52199433-5919-4c03-8b52-3322c1ebac0d None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:53.313120 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-52199433-5919-4c03-8b52-3322c1ebac0d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:59:53.313120 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:53.313120 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:53.313120 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:53.313120 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:53.313120 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:53.313120 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:53.313120 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:53.313120 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:53.313120 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:53.313120 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:53.313120 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:53.313120 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 04:59:53.313120 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 04:59:53.313120 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 04:59:53.314014 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 04:59:53.314014 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:53.314014 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:53.314014 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:53.314014 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:53.314014 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:53.314014 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:53.314014 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:53.314014 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:53.314014 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:53.314014 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:53.314014 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:53.314014 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:53.314014 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:53.314014 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:53.314014 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:53.314014 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:53.314928 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 04:59:53.314928 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 04:59:53.314928 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 04:59:53.314928 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 04:59:53.314928 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 04:59:53.314928 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 04:59:53.314928 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:59:53.314928 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 04:59:53.317458 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 350/690] 10.222.0.77 () {56 vars in 1111 bytes} [Tue Aug 30 04:59:53 2022] GET /identity/v3/projects/service => generated 88 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:53.324630 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52199433-5919-4c03-8b52-3322c1ebac0d None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:53.352962 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:53.437420 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52199433-5919-4c03-8b52-3322c1ebac0d None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:53.445480 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bf50ea5-7659-4ba5-adf6-791ef4a014e2 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:53.477526 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a224b078-45ea-4ad7-afff-06454fc014ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:53.478591 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a224b078-45ea-4ad7-afff-06454fc014ae None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:53.479004 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a224b078-45ea-4ad7-afff-06454fc014ae None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:53.479472 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a224b078-45ea-4ad7-afff-06454fc014ae None admin] PATH_INFO: `/v3/projects` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:53.480274 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a224b078-45ea-4ad7-afff-06454fc014ae None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:53.480643 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a224b078-45ea-4ad7-afff-06454fc014ae None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:53.482225 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a224b078-45ea-4ad7-afff-06454fc014ae None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:53.495828 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 351/691] 10.222.0.77 () {56 vars in 1148 bytes} [Tue Aug 30 04:59:53 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:53.501725 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a224b078-45ea-4ad7-afff-06454fc014ae None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:53.584180 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9881c293-ff32-4aaa-bea8-e1a8ed3510f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:53.584768 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9881c293-ff32-4aaa-bea8-e1a8ed3510f5 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:53.585008 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9881c293-ff32-4aaa-bea8-e1a8ed3510f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:53.585905 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9881c293-ff32-4aaa-bea8-e1a8ed3510f5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:53.595190 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:53.612138 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a224b078-45ea-4ad7-afff-06454fc014ae None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:53.615627 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9881c293-ff32-4aaa-bea8-e1a8ed3510f5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:53.617621 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9881c293-ff32-4aaa-bea8-e1a8ed3510f5 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:53.649921 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9d89e0f-8185-42e0-9516-c329d189fdc4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:53.650518 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9d89e0f-8185-42e0-9516-c329d189fdc4 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:53.650794 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9d89e0f-8185-42e0-9516-c329d189fdc4 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:53.651088 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9d89e0f-8185-42e0-9516-c329d189fdc4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:53.651748 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-9881c293-ff32-4aaa-bea8-e1a8ed3510f5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:59:53.651748 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:53.651748 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:53.651748 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:53.651748 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:53.651748 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:53.651748 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:53.651748 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:53.651748 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:53.651748 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:53.651748 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:53.651748 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:53.651748 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:59:53.651748 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:59:53.651748 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:59:53.651748 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:59:53.652658 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:53.652658 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:53.652658 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:53.652658 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:53.652658 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:53.652658 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:53.652658 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:53.652658 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:53.652658 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:53.652658 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:53.652658 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:53.652658 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:53.652658 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:53.652658 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:53.652658 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:53.652658 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:53.652658 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:59:53.653545 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:59:53.653545 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:59:53.653545 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 04:59:53.653732 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9d89e0f-8185-42e0-9516-c329d189fdc4 None admin] RBAC: Adding query filter params (role.id=821647e8b40542499667e29e1b301cb2, scope.project.id=e9919141e3b947ab8d91b55bba4a7de6, user.id=e0a7f7afc3d44ec788c05d005e52a1b8) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:53.653732 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9d89e0f-8185-42e0-9516-c329d189fdc4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:53.653732 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9d89e0f-8185-42e0-9516-c329d189fdc4 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:53.657337 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 341/692] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 04:59:53 2022] GET /identity/v3/domains/Default => generated 87 bytes in 487 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:53.658406 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9881c293-ff32-4aaa-bea8-e1a8ed3510f5 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:53.660171 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 352/693] 10.222.0.77 () {56 vars in 1374 bytes} [Tue Aug 30 04:59:53 2022] GET /identity/v3/role_assignments?user.id=e0a7f7afc3d44ec788c05d005e52a1b8&scope.project.id=e9919141e3b947ab8d91b55bba4a7de6&role.id=821647e8b40542499667e29e1b301cb2 => generated 608 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:59:53.777825 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:53.852326 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9881c293-ff32-4aaa-bea8-e1a8ed3510f5 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:53.889319 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47c6a668-8ffd-480d-b21f-4af364ef893b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:53.889944 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c6a668-8ffd-480d-b21f-4af364ef893b None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:53.890220 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c6a668-8ffd-480d-b21f-4af364ef893b None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:53.890487 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c6a668-8ffd-480d-b21f-4af364ef893b None admin] PATH_INFO: `/v3/domains` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:53.891172 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47c6a668-8ffd-480d-b21f-4af364ef893b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:53.891471 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47c6a668-8ffd-480d-b21f-4af364ef893b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:53.892880 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47c6a668-8ffd-480d-b21f-4af364ef893b None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:53.906632 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 342/694] 10.222.0.77 () {56 vars in 1109 bytes} [Tue Aug 30 04:59:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 248 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:53.912722 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9d89e0f-8185-42e0-9516-c329d189fdc4 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:54.020250 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9d89e0f-8185-42e0-9516-c329d189fdc4 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:54.055511 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a421b79c-b384-4787-b6b5-f3d95be361a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:54.056116 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a421b79c-b384-4787-b6b5-f3d95be361a8 None admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:54.056392 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a421b79c-b384-4787-b6b5-f3d95be361a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:54.056661 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a421b79c-b384-4787-b6b5-f3d95be361a8 None admin] PATH_INFO: `/v3/users` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:54.057295 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a421b79c-b384-4787-b6b5-f3d95be361a8 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:54.058725 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a421b79c-b384-4787-b6b5-f3d95be361a8 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:54.084321 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 353/695] 10.222.0.77 () {60 vars in 1132 bytes} [Tue Aug 30 04:59:53 2022] POST /identity/v3/users => generated 254 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:59:54.333108 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9410d463-6d85-4930-b9ea-a1fe833434b4 None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:54.333524 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9410d463-6d85-4930-b9ea-a1fe833434b4 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:54.333638 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9410d463-6d85-4930-b9ea-a1fe833434b4 None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:54.335104 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 343/696] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:59:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:59:54.341343 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-343ff13b-a3cc-49b7-be2f-98f855a92f28 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:54.341887 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-343ff13b-a3cc-49b7-be2f-98f855a92f28 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:54.342359 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-343ff13b-a3cc-49b7-be2f-98f855a92f28 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:54.350635 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:54.365742 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-343ff13b-a3cc-49b7-be2f-98f855a92f28 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:59:54.368759 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-343ff13b-a3cc-49b7-be2f-98f855a92f28 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:59:54.457844 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 354/697] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:59:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:59:54.682643 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e517240a-3dd6-4db6-b3a8-a56b919f4387 None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:54.683163 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e517240a-3dd6-4db6-b3a8-a56b919f4387 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:54.683163 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e517240a-3dd6-4db6-b3a8-a56b919f4387 None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:54.684723 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 344/698] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:59:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:59:54.689766 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f83aa889-bc6b-4327-b8ef-99bf76818f9d None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:54.690062 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f83aa889-bc6b-4327-b8ef-99bf76818f9d None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:54.690404 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f83aa889-bc6b-4327-b8ef-99bf76818f9d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:54.715902 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-f83aa889-bc6b-4327-b8ef-99bf76818f9d None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:59:54.717126 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-f83aa889-bc6b-4327-b8ef-99bf76818f9d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:59:54.804294 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 355/699] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:59:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:59:54.819041 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e517240a-3dd6-4db6-b3a8-a56b919f4387 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:54.895150 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2175aab6-ab88-490d-ad8c-9ba10e603154 None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:54.895465 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2175aab6-ab88-490d-ad8c-9ba10e603154 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:54.895715 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2175aab6-ab88-490d-ad8c-9ba10e603154 None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:54.897457 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 356/700] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:59:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:59:54.904583 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34226b04-0960-4691-b725-1c717b9395ed None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:54.905116 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34226b04-0960-4691-b725-1c717b9395ed None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:54.905680 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34226b04-0960-4691-b725-1c717b9395ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:54.938880 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-34226b04-0960-4691-b725-1c717b9395ed None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:59:54.940084 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-34226b04-0960-4691-b725-1c717b9395ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:59:54.959753 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e517240a-3dd6-4db6-b3a8-a56b919f4387 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:54.974438 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:55.002963 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74ef1980-825a-4a62-a781-0ae8996657fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:55.004084 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74ef1980-825a-4a62-a781-0ae8996657fb None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:55.004567 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74ef1980-825a-4a62-a781-0ae8996657fb None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:55.005288 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74ef1980-825a-4a62-a781-0ae8996657fb None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:55.006547 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74ef1980-825a-4a62-a781-0ae8996657fb None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:55.008719 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74ef1980-825a-4a62-a781-0ae8996657fb None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:55.020769 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-74ef1980-825a-4a62-a781-0ae8996657fb None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 30 04:59:55.020769 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:55.020769 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 30 04:59:55.020769 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 30 04:59:55.020769 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 30 04:59:55.020769 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 30 04:59:55.020769 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 30 04:59:55.020769 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 04:59:55.020769 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 30 04:59:55.020769 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 04:59:55.020769 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 04:59:55.020769 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 04:59:55.020769 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:55.020769 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:55.020769 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:55.020769 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:55.020769 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:55.021849 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:55.021849 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:55.021849 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:55.021849 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:55.021849 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:55.021849 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:55.021849 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 30 04:59:55.021849 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 30 04:59:55.021849 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 30 04:59:55.021849 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 30 04:59:55.021849 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:55.021849 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:55.021849 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:55.021849 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:55.021849 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:55.021849 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:55.022693 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:55.022693 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:55.022693 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:55.022693 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:55.022693 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:55.022693 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:55.022693 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:55.022693 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:55.022693 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:55.022693 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:55.022693 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 30 04:59:55.022693 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 04:59:55.022693 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 30 04:59:55.022693 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 04:59:55.026984 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 345/701] 10.222.0.77 () {56 vars in 1129 bytes} [Tue Aug 30 04:59:54 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:55.038882 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74ef1980-825a-4a62-a781-0ae8996657fb None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:55.067579 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:55.096522 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 357/702] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:59:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:59:55.097551 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34226b04-0960-4691-b725-1c717b9395ed None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:55.156385 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74ef1980-825a-4a62-a781-0ae8996657fb None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:55.194643 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af9e47af-630c-480f-8db9-03f123bcf3c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:55.195768 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af9e47af-630c-480f-8db9-03f123bcf3c8 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:55.196044 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af9e47af-630c-480f-8db9-03f123bcf3c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:55.196311 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af9e47af-630c-480f-8db9-03f123bcf3c8 None admin] PATH_INFO: `/v3/services` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:55.196928 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af9e47af-630c-480f-8db9-03f123bcf3c8 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:55.197226 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af9e47af-630c-480f-8db9-03f123bcf3c8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:55.198596 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af9e47af-630c-480f-8db9-03f123bcf3c8 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:55.206719 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 346/703] 10.222.0.77 () {56 vars in 1124 bytes} [Tue Aug 30 04:59:55 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:55.212228 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af9e47af-630c-480f-8db9-03f123bcf3c8 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:55.278829 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:55.300092 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34226b04-0960-4691-b725-1c717b9395ed None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:55.348799 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffb2f01c-f877-4485-9e51-ba66beea645b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:55.349580 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb2f01c-f877-4485-9e51-ba66beea645b None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:55.349864 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb2f01c-f877-4485-9e51-ba66beea645b None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:55.350135 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb2f01c-f877-4485-9e51-ba66beea645b None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:55.361667 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffb2f01c-f877-4485-9e51-ba66beea645b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:59:55.367838 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-ffb2f01c-f877-4485-9e51-ba66beea645b None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 04:59:55.367838 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:55.367838 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:55.367838 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:55.367838 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:55.367838 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:55.367838 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:55.367838 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:55.367838 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:55.367838 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:55.367838 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:55.367838 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:55.367838 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:59:55.367838 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:59:55.367838 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:59:55.367838 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:59:55.369220 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:59:55.369220 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:59:55.369220 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:59:55.369220 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:59:55.369220 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:55.369220 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:55.369220 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:55.369220 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:55.369220 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:55.369220 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:55.369220 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:55.369220 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:55.369220 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:55.369220 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:55.369220 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:55.369220 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:55.370129 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:55.370129 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:55.370129 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:55.370129 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:55.370129 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:59:55.370129 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:59:55.370129 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:59:55.370129 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:59:55.370129 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:59:55.370129 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:59:55.370129 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 04:59:55.370129 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 04:59:55.370129 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 358/704] 10.222.0.77 () {56 vars in 1096 bytes} [Tue Aug 30 04:59:55 2022] GET /identity/v3/roles/admin => generated 83 bytes in 272 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:55.370836 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffb2f01c-f877-4485-9e51-ba66beea645b None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:55.386569 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af9e47af-630c-480f-8db9-03f123bcf3c8 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:55.424144 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ad1d3b1-6bae-4745-af98-01df32b506ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:55.424861 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ad1d3b1-6bae-4745-af98-01df32b506ea None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:55.425200 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ad1d3b1-6bae-4745-af98-01df32b506ea None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:55.425538 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ad1d3b1-6bae-4745-af98-01df32b506ea None admin] PATH_INFO: `/v3/services` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:55.426229 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ad1d3b1-6bae-4745-af98-01df32b506ea None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:55.426595 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ad1d3b1-6bae-4745-af98-01df32b506ea None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:55.428080 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ad1d3b1-6bae-4745-af98-01df32b506ea None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:55.435380 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 347/705] 10.222.0.77 () {56 vars in 1124 bytes} [Tue Aug 30 04:59:55 2022] GET /identity/v3/services?type=block-storage => generated 130 bytes in 223 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:59:55.442396 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ad1d3b1-6bae-4745-af98-01df32b506ea None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:55.444007 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:55.504503 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffb2f01c-f877-4485-9e51-ba66beea645b None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:55.547747 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71ae7fd7-835d-445c-ab5d-06489a6bb6a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:55.548843 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ae7fd7-835d-445c-ab5d-06489a6bb6a4 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:55.549299 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ae7fd7-835d-445c-ab5d-06489a6bb6a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:55.549779 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ae7fd7-835d-445c-ab5d-06489a6bb6a4 None admin] PATH_INFO: `/v3/services/network` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:55.550639 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71ae7fd7-835d-445c-ab5d-06489a6bb6a4 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:55.552298 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71ae7fd7-835d-445c-ab5d-06489a6bb6a4 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:55.565939 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-71ae7fd7-835d-445c-ab5d-06489a6bb6a4 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Aug 30 04:59:55.565939 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:55.565939 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 30 04:59:55.565939 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 30 04:59:55.565939 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 30 04:59:55.565939 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 30 04:59:55.565939 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 30 04:59:55.565939 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 04:59:55.565939 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Aug 30 04:59:55.565939 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 04:59:55.565939 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 04:59:55.565939 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 04:59:55.565939 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:55.565939 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:55.565939 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:55.565939 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:55.565939 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:55.567029 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:55.567029 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:55.567029 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:55.567029 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:55.567029 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:55.567029 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:55.567029 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 30 04:59:55.567029 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 30 04:59:55.567029 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 30 04:59:55.567029 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 30 04:59:55.567029 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:55.567029 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:55.567029 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:55.567029 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:55.567029 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:55.567029 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:55.567858 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:55.567858 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:55.567858 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:55.567858 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:55.567858 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:55.567858 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:55.567858 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:55.567858 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:55.567858 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:55.567858 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:55.567858 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 30 04:59:55.567858 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 04:59:55.567858 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Aug 30 04:59:55.567858 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 04:59:55.570639 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 359/706] 10.222.0.77 () {56 vars in 1111 bytes} [Tue Aug 30 04:59:55 2022] GET /identity/v3/services/network => generated 88 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:55.576644 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71ae7fd7-835d-445c-ab5d-06489a6bb6a4 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:55.679682 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ad1d3b1-6bae-4745-af98-01df32b506ea None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:55.680339 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:55.739273 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89687426-6f4f-493e-a313-c43874fcbf64 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:55.741156 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89687426-6f4f-493e-a313-c43874fcbf64 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:55.744920 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89687426-6f4f-493e-a313-c43874fcbf64 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:55.745336 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89687426-6f4f-493e-a313-c43874fcbf64 None admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:55.746169 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89687426-6f4f-493e-a313-c43874fcbf64 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:55.746611 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89687426-6f4f-493e-a313-c43874fcbf64 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:55.748142 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89687426-6f4f-493e-a313-c43874fcbf64 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:55.761658 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 348/707] 10.222.0.77 () {56 vars in 1099 bytes} [Tue Aug 30 04:59:55 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 326 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:55.768525 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71ae7fd7-835d-445c-ab5d-06489a6bb6a4 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:55.769531 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89687426-6f4f-493e-a313-c43874fcbf64 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:55.804263 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:55.836446 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfb09b29-47fb-46bd-b1cf-f9ff8451b36a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:55.837206 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfb09b29-47fb-46bd-b1cf-f9ff8451b36a None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:55.837594 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfb09b29-47fb-46bd-b1cf-f9ff8451b36a None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:55.837936 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfb09b29-47fb-46bd-b1cf-f9ff8451b36a None admin] PATH_INFO: `/v3/services` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:55.838892 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfb09b29-47fb-46bd-b1cf-f9ff8451b36a None admin] RBAC: Adding query filter params (name=network) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:55.839094 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfb09b29-47fb-46bd-b1cf-f9ff8451b36a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:55.840545 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfb09b29-47fb-46bd-b1cf-f9ff8451b36a None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:55.854708 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 360/708] 10.222.0.77 () {56 vars in 1112 bytes} [Tue Aug 30 04:59:55 2022] GET /identity/v3/services?name=network => generated 124 bytes in 278 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:55.859815 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfb09b29-47fb-46bd-b1cf-f9ff8451b36a None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:55.883823 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89687426-6f4f-493e-a313-c43874fcbf64 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:55.920269 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c6c4348-83b0-4059-84d2-93aa69d2acdf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:55.921201 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c6c4348-83b0-4059-84d2-93aa69d2acdf None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:55.921705 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c6c4348-83b0-4059-84d2-93aa69d2acdf None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:55.922174 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c6c4348-83b0-4059-84d2-93aa69d2acdf None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:55.933635 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c6c4348-83b0-4059-84d2-93aa69d2acdf None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:55.935414 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c6c4348-83b0-4059-84d2-93aa69d2acdf None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:55.950417 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-6c6c4348-83b0-4059-84d2-93aa69d2acdf None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:59:55.950417 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:55.950417 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:55.950417 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:55.950417 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:55.950417 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:55.950417 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:55.950417 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:55.950417 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:55.950417 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:55.950417 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:55.950417 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:55.950417 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:59:55.950417 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:59:55.950417 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:59:55.950417 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:59:55.951438 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:55.951438 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:55.951438 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:55.951438 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:55.951438 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:55.951438 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:55.951438 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:55.951438 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:55.951438 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:55.951438 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:55.951438 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:55.951438 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:55.951438 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:55.951438 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:55.951438 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:55.951438 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:55.951438 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:59:55.952313 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:59:55.952313 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:59:55.952313 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 04:59:55.954434 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 349/709] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 04:59:55 2022] GET /identity/v3/domains/Default => generated 87 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:55.961355 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c6c4348-83b0-4059-84d2-93aa69d2acdf None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:55.991292 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:56.072331 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c6c4348-83b0-4059-84d2-93aa69d2acdf None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:56.109044 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-760b7520-6613-4285-aceb-abe0dd49a9e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:56.110030 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-760b7520-6613-4285-aceb-abe0dd49a9e5 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:56.110507 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-760b7520-6613-4285-aceb-abe0dd49a9e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:56.111018 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-760b7520-6613-4285-aceb-abe0dd49a9e5 None admin] PATH_INFO: `/v3/domains` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:56.111849 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-760b7520-6613-4285-aceb-abe0dd49a9e5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:56.112361 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-760b7520-6613-4285-aceb-abe0dd49a9e5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:56.113921 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-760b7520-6613-4285-aceb-abe0dd49a9e5 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:56.116499 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfb09b29-47fb-46bd-b1cf-f9ff8451b36a None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:56.127354 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 350/710] 10.222.0.77 () {56 vars in 1109 bytes} [Tue Aug 30 04:59:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:59:56.133723 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-760b7520-6613-4285-aceb-abe0dd49a9e5 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:56.184148 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d082152b-e7cf-4931-b406-2987f8e50f17 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:56.185363 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d082152b-e7cf-4931-b406-2987f8e50f17 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:56.186781 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d082152b-e7cf-4931-b406-2987f8e50f17 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:56.187300 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d082152b-e7cf-4931-b406-2987f8e50f17 None admin] PATH_INFO: `/v3/services` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:56.188576 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d082152b-e7cf-4931-b406-2987f8e50f17 None admin] RBAC: Adding query filter params (type=network) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:56.189784 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d082152b-e7cf-4931-b406-2987f8e50f17 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:56.192477 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d082152b-e7cf-4931-b406-2987f8e50f17 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:56.205981 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 361/711] 10.222.0.77 () {56 vars in 1112 bytes} [Tue Aug 30 04:59:55 2022] GET /identity/v3/services?type=network => generated 124 bytes in 346 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:59:56.254329 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:56.285004 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-760b7520-6613-4285-aceb-abe0dd49a9e5 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:56.328241 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75526693-f5dc-44ec-a6b7-c2d76eab2301 None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:56.328909 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75526693-f5dc-44ec-a6b7-c2d76eab2301 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:56.329379 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75526693-f5dc-44ec-a6b7-c2d76eab2301 None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:56.334973 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 362/712] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:59:56 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 04:59:56.337496 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-466e3fbb-5400-4ba4-9593-a5a7ab441bf9 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:56.338152 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-466e3fbb-5400-4ba4-9593-a5a7ab441bf9 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:56.338503 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9427aab4-69cf-42ab-9ae8-cb44b499962c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:56.338923 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-466e3fbb-5400-4ba4-9593-a5a7ab441bf9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:56.339189 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9427aab4-69cf-42ab-9ae8-cb44b499962c None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:56.339471 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9427aab4-69cf-42ab-9ae8-cb44b499962c None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:56.339743 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9427aab4-69cf-42ab-9ae8-cb44b499962c None admin] PATH_INFO: `/v3/users/nova` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:56.351493 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9427aab4-69cf-42ab-9ae8-cb44b499962c None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:56.352975 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9427aab4-69cf-42ab-9ae8-cb44b499962c None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:56.361892 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-466e3fbb-5400-4ba4-9593-a5a7ab441bf9 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:59:56.363485 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-466e3fbb-5400-4ba4-9593-a5a7ab441bf9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:59:56.372514 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-9427aab4-69cf-42ab-9ae8-cb44b499962c None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 30 04:59:56.372514 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:56.372514 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:56.372514 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:56.372514 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:56.372514 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:56.372514 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:56.372514 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:56.372514 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:56.372514 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:56.372514 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:56.372514 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:56.372514 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 04:59:56.372514 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 04:59:56.372514 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 04:59:56.372514 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 04:59:56.373512 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:56.373512 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:56.373512 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 04:59:56.373512 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:59:56.373512 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 04:59:56.373512 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:59:56.373512 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:56.373512 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:56.373512 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:56.373512 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:56.373512 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:56.373512 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:56.373512 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:56.373512 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:56.373512 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:56.373512 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:56.373512 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:56.373512 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:56.374442 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:56.374442 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:56.374442 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 04:59:56.374442 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 04:59:56.374442 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 04:59:56.374442 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 04:59:56.374442 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 04:59:56.374442 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 04:59:56.374442 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 30 04:59:56.374442 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 04:59:56.375579 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 351/713] 10.222.0.77 () {56 vars in 1093 bytes} [Tue Aug 30 04:59:56 2022] GET /identity/v3/users/nova => generated 82 bytes in 242 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 04:59:56.389684 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9427aab4-69cf-42ab-9ae8-cb44b499962c None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:56.430494 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:56.462933 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 363/714] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:59:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 30 04:59:56.517977 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9427aab4-69cf-42ab-9ae8-cb44b499962c None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:56.553275 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6433bf4b-63af-473b-aa6c-79d1ff6f39ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:56.553895 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6433bf4b-63af-473b-aa6c-79d1ff6f39ec None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:56.554218 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6433bf4b-63af-473b-aa6c-79d1ff6f39ec None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:56.554488 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6433bf4b-63af-473b-aa6c-79d1ff6f39ec None admin] PATH_INFO: `/v3/users` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:56.555239 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6433bf4b-63af-473b-aa6c-79d1ff6f39ec None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:56.555537 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6433bf4b-63af-473b-aa6c-79d1ff6f39ec None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:56.556999 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6433bf4b-63af-473b-aa6c-79d1ff6f39ec None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:56.559666 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:56.577021 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 352/715] 10.222.0.77 () {56 vars in 1133 bytes} [Tue Aug 30 04:59:56 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:59:56.583049 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-466e3fbb-5400-4ba4-9593-a5a7ab441bf9 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:56.680719 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff7101e8-eb6e-420c-82e7-3b651b81825a None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:56.681044 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff7101e8-eb6e-420c-82e7-3b651b81825a None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:56.681652 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff7101e8-eb6e-420c-82e7-3b651b81825a None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:56.684150 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 353/716] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:59:56 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:59:56.693304 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63006ef2-dee2-4e23-99e1-c6e3655e5f48 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:56.693582 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63006ef2-dee2-4e23-99e1-c6e3655e5f48 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:56.693849 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-466e3fbb-5400-4ba4-9593-a5a7ab441bf9 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:56.693959 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63006ef2-dee2-4e23-99e1-c6e3655e5f48 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:56.713306 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-63006ef2-dee2-4e23-99e1-c6e3655e5f48 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:59:56.714616 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-63006ef2-dee2-4e23-99e1-c6e3655e5f48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:59:56.737103 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a7dcf80-eeb6-44d8-b587-e086974b6b88 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:56.737722 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a7dcf80-eeb6-44d8-b587-e086974b6b88 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:56.737996 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a7dcf80-eeb6-44d8-b587-e086974b6b88 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:56.738262 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a7dcf80-eeb6-44d8-b587-e086974b6b88 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:56.816158 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:56.821593 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 354/717] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:59:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:59:56.842965 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a7dcf80-eeb6-44d8-b587-e086974b6b88 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:56.844545 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a7dcf80-eeb6-44d8-b587-e086974b6b88 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:56.898639 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63006ef2-dee2-4e23-99e1-c6e3655e5f48 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:57.024111 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63006ef2-dee2-4e23-99e1-c6e3655e5f48 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:57.032048 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 364/718] 10.222.0.77 () {58 vars in 1282 bytes} [Tue Aug 30 04:59:56 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 450 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 04:59:57.037686 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a7dcf80-eeb6-44d8-b587-e086974b6b88 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:57.071237 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:57.081846 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0df650d6-e60a-4596-8863-bd52be933cd2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:57.082473 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0df650d6-e60a-4596-8863-bd52be933cd2 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:57.084259 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0df650d6-e60a-4596-8863-bd52be933cd2 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:57.084588 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0df650d6-e60a-4596-8863-bd52be933cd2 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:57.112779 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0df650d6-e60a-4596-8863-bd52be933cd2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:59:57.118087 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-0df650d6-e60a-4596-8863-bd52be933cd2 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 04:59:57.118087 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:57.118087 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:57.118087 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:57.118087 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:57.118087 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:57.118087 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:57.118087 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:57.118087 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:57.118087 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:57.118087 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:57.118087 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:57.118087 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:59:57.118087 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:59:57.118087 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:59:57.118087 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:59:57.119080 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:59:57.119080 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:59:57.119080 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:59:57.119080 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:59:57.119080 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:57.119080 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:57.119080 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:57.119080 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:57.119080 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:57.119080 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:57.119080 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:57.119080 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:57.119080 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:57.119080 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:57.119080 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:57.119080 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:57.119931 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:57.119931 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:57.119931 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:57.119931 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:57.119931 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:59:57.119931 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:59:57.119931 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:59:57.119931 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:59:57.119931 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:59:57.119931 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:59:57.119931 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 04:59:57.119931 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 04:59:57.123423 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 355/719] 10.222.0.77 () {56 vars in 1096 bytes} [Tue Aug 30 04:59:56 2022] GET /identity/v3/roles/admin => generated 83 bytes in 225 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 04:59:57.127590 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0df650d6-e60a-4596-8863-bd52be933cd2 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:57.170119 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:57.194241 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a7dcf80-eeb6-44d8-b587-e086974b6b88 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:57.230371 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70816951-ea5a-4d08-a44c-b25de6b505b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:57.231018 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70816951-ea5a-4d08-a44c-b25de6b505b3 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:57.231294 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70816951-ea5a-4d08-a44c-b25de6b505b3 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:57.231568 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70816951-ea5a-4d08-a44c-b25de6b505b3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:57.242250 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70816951-ea5a-4d08-a44c-b25de6b505b3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:57.243766 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70816951-ea5a-4d08-a44c-b25de6b505b3 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:57.253786 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0df650d6-e60a-4596-8863-bd52be933cd2 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:57.262487 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-70816951-ea5a-4d08-a44c-b25de6b505b3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:59:57.262487 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:57.262487 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:57.262487 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:57.262487 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:57.262487 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:57.262487 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:57.262487 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:57.262487 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:57.262487 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:57.262487 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:57.262487 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:57.262487 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:59:57.262487 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:59:57.262487 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:59:57.262487 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:59:57.264722 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:57.264722 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:57.264722 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:57.264722 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:57.264722 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:57.264722 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:57.264722 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:57.264722 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:57.264722 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:57.264722 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:57.264722 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:57.264722 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:57.264722 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:57.264722 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:57.264722 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:57.264722 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:57.264722 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:59:57.267058 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:59:57.267058 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:59:57.267058 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 04:59:57.267058 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 365/720] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 04:59:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 226 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:57.267058 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70816951-ea5a-4d08-a44c-b25de6b505b3 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:57.292369 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81da6aba-3205-4765-97cb-adf0b023331d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:57.293009 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81da6aba-3205-4765-97cb-adf0b023331d None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:57.293337 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81da6aba-3205-4765-97cb-adf0b023331d None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:57.293617 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81da6aba-3205-4765-97cb-adf0b023331d None admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:57.294282 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81da6aba-3205-4765-97cb-adf0b023331d None admin] RBAC: Adding query filter params (name=admin) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:57.294585 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81da6aba-3205-4765-97cb-adf0b023331d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:57.296233 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81da6aba-3205-4765-97cb-adf0b023331d None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:57.306541 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 356/721] 10.222.0.77 () {56 vars in 1099 bytes} [Tue Aug 30 04:59:57 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:57.307490 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81da6aba-3205-4765-97cb-adf0b023331d None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:57.360845 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:57.373334 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70816951-ea5a-4d08-a44c-b25de6b505b3 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:57.403281 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:57.410963 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-527db5f6-3805-4b56-9bde-a723fef1cc52 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:57.411693 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-527db5f6-3805-4b56-9bde-a723fef1cc52 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:57.412132 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-527db5f6-3805-4b56-9bde-a723fef1cc52 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:57.412615 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-527db5f6-3805-4b56-9bde-a723fef1cc52 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:57.425864 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81da6aba-3205-4765-97cb-adf0b023331d None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:57.426339 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-527db5f6-3805-4b56-9bde-a723fef1cc52 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:59:57.432970 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-527db5f6-3805-4b56-9bde-a723fef1cc52 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 30 04:59:57.432970 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:57.432970 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:57.432970 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:57.432970 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:57.432970 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:57.432970 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:57.432970 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:57.432970 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:57.432970 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:57.432970 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:57.432970 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:57.432970 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:59:57.432970 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:59:57.432970 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:59:57.432970 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:59:57.434348 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:59:57.434348 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:59:57.434348 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:59:57.434348 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:59:57.434348 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:57.434348 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:57.434348 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:57.434348 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:57.434348 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:57.434348 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:57.434348 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:57.434348 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:57.434348 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:57.434348 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:57.434348 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:57.434348 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:57.435233 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:57.435233 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:57.435233 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:57.435233 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:57.435233 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:59:57.435233 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:59:57.435233 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:59:57.435233 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:59:57.435233 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:59:57.435233 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:59:57.435233 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 30 04:59:57.435233 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 04:59:57.438907 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 366/722] 10.222.0.77 () {56 vars in 1102 bytes} [Tue Aug 30 04:59:57 2022] GET /identity/v3/roles/service => generated 85 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:57.441201 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-527db5f6-3805-4b56-9bde-a723fef1cc52 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:57.478961 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22594ce8-046e-4df3-bb10-ee812763681d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:57.479835 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22594ce8-046e-4df3-bb10-ee812763681d None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:57.480345 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22594ce8-046e-4df3-bb10-ee812763681d None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:57.480908 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22594ce8-046e-4df3-bb10-ee812763681d None admin] PATH_INFO: `/v3/domains` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:57.481869 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22594ce8-046e-4df3-bb10-ee812763681d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:57.482355 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22594ce8-046e-4df3-bb10-ee812763681d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:57.484012 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22594ce8-046e-4df3-bb10-ee812763681d None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:57.502206 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 357/723] 10.222.0.77 () {56 vars in 1109 bytes} [Tue Aug 30 04:59:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:57.503427 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22594ce8-046e-4df3-bb10-ee812763681d None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:57.571153 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-527db5f6-3805-4b56-9bde-a723fef1cc52 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:57.610585 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6b7fef1-28a2-499d-b77e-024fc25833de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:57.611327 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b7fef1-28a2-499d-b77e-024fc25833de None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:57.611550 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b7fef1-28a2-499d-b77e-024fc25833de None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:57.611824 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b7fef1-28a2-499d-b77e-024fc25833de None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:57.626356 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6b7fef1-28a2-499d-b77e-024fc25833de None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:57.626630 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22594ce8-046e-4df3-bb10-ee812763681d None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:57.628012 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6b7fef1-28a2-499d-b77e-024fc25833de None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:57.635483 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:57.649471 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-a6b7fef1-28a2-499d-b77e-024fc25833de None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:59:57.649471 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:57.649471 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:57.649471 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:57.649471 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:57.649471 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:57.649471 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:57.649471 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:57.649471 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:57.649471 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:57.649471 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:57.649471 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:57.649471 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:59:57.649471 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:59:57.649471 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:59:57.649471 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:59:57.650439 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:57.650439 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:57.650439 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:57.650439 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:57.650439 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:57.650439 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:57.650439 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:57.650439 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:57.650439 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:57.650439 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:57.650439 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:57.650439 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:57.650439 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:57.650439 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:57.650439 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:57.650439 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:57.650439 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:59:57.651382 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:59:57.651382 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:59:57.651382 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 04:59:57.651382 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 367/724] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 04:59:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:57.652048 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6b7fef1-28a2-499d-b77e-024fc25833de None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:57.669415 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d72e6dd3-5655-4167-8725-9d00281cbbdf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:57.670375 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d72e6dd3-5655-4167-8725-9d00281cbbdf None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:57.670930 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d72e6dd3-5655-4167-8725-9d00281cbbdf None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:57.671484 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d72e6dd3-5655-4167-8725-9d00281cbbdf None admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:57.672478 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d72e6dd3-5655-4167-8725-9d00281cbbdf None admin] RBAC: Adding query filter params (name=service) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:57.672994 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d72e6dd3-5655-4167-8725-9d00281cbbdf None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:57.674553 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d72e6dd3-5655-4167-8725-9d00281cbbdf None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:57.684167 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 358/725] 10.222.0.77 () {56 vars in 1103 bytes} [Tue Aug 30 04:59:57 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:57.685198 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d72e6dd3-5655-4167-8725-9d00281cbbdf None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:57.710025 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:57.779976 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6b7fef1-28a2-499d-b77e-024fc25833de None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:57.800013 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d72e6dd3-5655-4167-8725-9d00281cbbdf None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:57.817962 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-035c78e0-2396-4131-b656-226443ad3905 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:57.818933 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-035c78e0-2396-4131-b656-226443ad3905 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:57.819408 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-035c78e0-2396-4131-b656-226443ad3905 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:57.820415 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-035c78e0-2396-4131-b656-226443ad3905 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:57.828679 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-035c78e0-2396-4131-b656-226443ad3905 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:57.830124 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-035c78e0-2396-4131-b656-226443ad3905 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:57.836943 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c61b1cc-746e-47cc-b8a0-b7177b4ef051 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:57.837756 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c61b1cc-746e-47cc-b8a0-b7177b4ef051 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:57.838154 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c61b1cc-746e-47cc-b8a0-b7177b4ef051 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:57.838430 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c61b1cc-746e-47cc-b8a0-b7177b4ef051 None admin] PATH_INFO: `/v3/domains` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:57.839091 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c61b1cc-746e-47cc-b8a0-b7177b4ef051 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:57.839388 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c61b1cc-746e-47cc-b8a0-b7177b4ef051 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:57.840730 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c61b1cc-746e-47cc-b8a0-b7177b4ef051 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:57.842927 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-035c78e0-2396-4131-b656-226443ad3905 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:59:57.842927 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:57.842927 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:57.842927 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:57.842927 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:57.842927 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:57.842927 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:57.842927 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:57.842927 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:57.842927 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:57.842927 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:57.842927 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:57.842927 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 04:59:57.842927 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 04:59:57.842927 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 04:59:57.844872 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 04:59:57.844872 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:57.844872 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:57.844872 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:57.844872 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:57.844872 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:57.844872 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:57.844872 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:57.844872 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:57.844872 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:57.844872 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:57.844872 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:57.844872 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:57.844872 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:57.844872 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:57.844872 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:57.844872 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:57.847161 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 04:59:57.847161 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 04:59:57.847161 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 04:59:57.847161 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 04:59:57.847161 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 04:59:57.847161 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 04:59:57.847161 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:59:57.847161 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 04:59:57.847161 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 368/726] 10.222.0.77 () {56 vars in 1111 bytes} [Tue Aug 30 04:59:57 2022] GET /identity/v3/projects/service => generated 88 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:57.847161 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-035c78e0-2396-4131-b656-226443ad3905 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:57.859978 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 359/727] 10.222.0.77 () {56 vars in 1109 bytes} [Tue Aug 30 04:59:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:57.861173 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c16deba-1cc6-4e61-9ef9-229452f76f3e None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:57.861505 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c16deba-1cc6-4e61-9ef9-229452f76f3e None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:57.861723 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c16deba-1cc6-4e61-9ef9-229452f76f3e None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:57.867031 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 360/728] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:59:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:59:57.867031 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c16deba-1cc6-4e61-9ef9-229452f76f3e None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:57.896416 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:57.951414 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:57.955019 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-035c78e0-2396-4131-b656-226443ad3905 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:57.979992 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c16deba-1cc6-4e61-9ef9-229452f76f3e None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:57.990335 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04a446bf-df78-4df4-b817-1777b7c5bbbe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:57.990987 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04a446bf-df78-4df4-b817-1777b7c5bbbe None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:57.991264 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04a446bf-df78-4df4-b817-1777b7c5bbbe None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:57.991563 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04a446bf-df78-4df4-b817-1777b7c5bbbe None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:58.002098 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04a446bf-df78-4df4-b817-1777b7c5bbbe None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:58.003547 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04a446bf-df78-4df4-b817-1777b7c5bbbe None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:58.017627 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-429b67e6-ea9c-4240-952a-83525f94e8a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:58.018099 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-04a446bf-df78-4df4-b817-1777b7c5bbbe None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:59:58.018099 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:58.018099 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:58.018099 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:58.018099 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:58.018099 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:58.018099 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:58.018099 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:58.018099 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:58.018099 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:58.018099 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:58.018099 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:58.018099 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:59:58.018099 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:59:58.018099 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:59:58.018099 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:59:58.019261 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:58.019261 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:58.019261 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:58.019261 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:58.019261 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:58.019261 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:58.019261 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:58.019261 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:58.019261 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:58.019261 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:58.019261 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:58.019261 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:58.019261 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:58.019261 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:58.019261 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:58.019261 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:58.019261 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:59:58.020393 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:59:58.020393 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:59:58.020393 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 04:59:58.021022 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-429b67e6-ea9c-4240-952a-83525f94e8a2 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:58.021444 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-429b67e6-ea9c-4240-952a-83525f94e8a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:58.021852 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-429b67e6-ea9c-4240-952a-83525f94e8a2 None admin] PATH_INFO: `/v3/projects` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:58.023305 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-429b67e6-ea9c-4240-952a-83525f94e8a2 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:58.023779 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-429b67e6-ea9c-4240-952a-83525f94e8a2 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:58.024438 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 369/729] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 04:59:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:58.025251 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04a446bf-df78-4df4-b817-1777b7c5bbbe None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:58.025907 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-429b67e6-ea9c-4240-952a-83525f94e8a2 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:58.039234 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 361/730] 10.222.0.77 () {56 vars in 1148 bytes} [Tue Aug 30 04:59:57 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:58.040474 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0b0701b-6df1-4c15-afd3-ff0dfd44f10d None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:58.040759 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0b0701b-6df1-4c15-afd3-ff0dfd44f10d None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:58.041036 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0b0701b-6df1-4c15-afd3-ff0dfd44f10d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:58.060162 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-f0b0701b-6df1-4c15-afd3-ff0dfd44f10d None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:59:58.061578 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-f0b0701b-6df1-4c15-afd3-ff0dfd44f10d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:59:58.133658 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04a446bf-df78-4df4-b817-1777b7c5bbbe None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:58.142887 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 362/731] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:59:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:59:58.142887 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0b0701b-6df1-4c15-afd3-ff0dfd44f10d None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:58.159260 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:58.176424 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c4cfaea-baaa-452f-a54a-016a1df2fc65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:58.177789 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c4cfaea-baaa-452f-a54a-016a1df2fc65 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:58.178076 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c4cfaea-baaa-452f-a54a-016a1df2fc65 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:58.178345 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c4cfaea-baaa-452f-a54a-016a1df2fc65 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:58.187666 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c4cfaea-baaa-452f-a54a-016a1df2fc65 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:58.189266 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c4cfaea-baaa-452f-a54a-016a1df2fc65 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:58.205858 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-0c4cfaea-baaa-452f-a54a-016a1df2fc65 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Aug 30 04:59:58.205858 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:58.205858 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:58.205858 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:58.205858 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:58.205858 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:58.205858 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:58.205858 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:58.205858 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:58.205858 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:58.205858 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:58.205858 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:58.205858 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 04:59:58.205858 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 04:59:58.205858 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 04:59:58.205858 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 04:59:58.207060 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:58.207060 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:58.207060 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 04:59:58.207060 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:59:58.207060 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 04:59:58.207060 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:59:58.207060 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:58.207060 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:58.207060 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:58.207060 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:58.207060 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:58.207060 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:58.207060 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:58.207060 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:58.207060 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:58.207060 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:58.207060 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:58.207060 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:58.208039 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:58.208039 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:58.208039 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 04:59:58.208039 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 04:59:58.208039 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 04:59:58.208039 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 04:59:58.208039 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 04:59:58.208039 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 04:59:58.208039 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Aug 30 04:59:58.208039 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 04:59:58.208039 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 370/732] 10.222.0.77 () {56 vars in 1096 bytes} [Tue Aug 30 04:59:58 2022] GET /identity/v3/users/swift => generated 83 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:58.208666 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c4cfaea-baaa-452f-a54a-016a1df2fc65 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:58.229020 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:58.254144 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0b0701b-6df1-4c15-afd3-ff0dfd44f10d None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:58.299600 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-224ca7ce-3ddc-473a-a2aa-3447ee540122 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:58.300306 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-224ca7ce-3ddc-473a-a2aa-3447ee540122 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:58.301497 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-224ca7ce-3ddc-473a-a2aa-3447ee540122 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:58.301848 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-224ca7ce-3ddc-473a-a2aa-3447ee540122 None admin] PATH_INFO: `/v3/domains` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:58.302588 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-224ca7ce-3ddc-473a-a2aa-3447ee540122 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:58.302976 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-224ca7ce-3ddc-473a-a2aa-3447ee540122 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:58.304943 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-224ca7ce-3ddc-473a-a2aa-3447ee540122 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:58.323143 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 363/733] 10.222.0.77 () {56 vars in 1109 bytes} [Tue Aug 30 04:59:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:58.324298 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-224ca7ce-3ddc-473a-a2aa-3447ee540122 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:58.338555 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c4cfaea-baaa-452f-a54a-016a1df2fc65 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:58.377014 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac6d608c-39bb-4702-ae9b-40a8350815e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:58.377616 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac6d608c-39bb-4702-ae9b-40a8350815e6 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:58.377880 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac6d608c-39bb-4702-ae9b-40a8350815e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:58.378458 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac6d608c-39bb-4702-ae9b-40a8350815e6 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:58.379204 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac6d608c-39bb-4702-ae9b-40a8350815e6 None admin] RBAC: Adding query filter params (role.id=9d9d5d2c270146488249b7ed943280db, scope.project.id=e9919141e3b947ab8d91b55bba4a7de6, user.id=886eb5e00bbf4ec292eb962b665b37d0) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:58.379519 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac6d608c-39bb-4702-ae9b-40a8350815e6 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:58.380920 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac6d608c-39bb-4702-ae9b-40a8350815e6 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:58.387003 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 371/734] 10.222.0.77 () {56 vars in 1374 bytes} [Tue Aug 30 04:59:58 2022] GET /identity/v3/role_assignments?user.id=886eb5e00bbf4ec292eb962b665b37d0&scope.project.id=e9919141e3b947ab8d91b55bba4a7de6&role.id=9d9d5d2c270146488249b7ed943280db => generated 259 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:58.387706 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac6d608c-39bb-4702-ae9b-40a8350815e6 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:58.426529 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:58.453668 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-224ca7ce-3ddc-473a-a2aa-3447ee540122 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:58.494440 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91c195e5-2ff2-43c0-bca9-4ba38bbaa2e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:58.495086 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c195e5-2ff2-43c0-bca9-4ba38bbaa2e1 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:58.495363 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c195e5-2ff2-43c0-bca9-4ba38bbaa2e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:58.495633 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c195e5-2ff2-43c0-bca9-4ba38bbaa2e1 None admin] PATH_INFO: `/v3/users` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:58.496368 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91c195e5-2ff2-43c0-bca9-4ba38bbaa2e1 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:58.496665 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91c195e5-2ff2-43c0-bca9-4ba38bbaa2e1 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:58.498113 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91c195e5-2ff2-43c0-bca9-4ba38bbaa2e1 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:58.498655 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:58.523466 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 364/735] 10.222.0.77 () {56 vars in 1135 bytes} [Tue Aug 30 04:59:58 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:58.524718 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d32e2112-7138-4c43-a18c-cacab94e2fc0 None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:58.525003 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d32e2112-7138-4c43-a18c-cacab94e2fc0 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:58.525275 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d32e2112-7138-4c43-a18c-cacab94e2fc0 None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:58.526696 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 365/736] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 04:59:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:59:58.531321 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac6d608c-39bb-4702-ae9b-40a8350815e6 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:58.531494 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d32e2112-7138-4c43-a18c-cacab94e2fc0 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:58.568730 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-477ea08f-e895-4749-be41-aebd922072d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:58.569354 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-477ea08f-e895-4749-be41-aebd922072d6 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:58.569672 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-477ea08f-e895-4749-be41-aebd922072d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:58.569940 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-477ea08f-e895-4749-be41-aebd922072d6 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:58.578305 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-477ea08f-e895-4749-be41-aebd922072d6 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:58.579795 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-477ea08f-e895-4749-be41-aebd922072d6 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:58.591257 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-477ea08f-e895-4749-be41-aebd922072d6 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 30 04:59:58.591257 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:58.591257 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:58.591257 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:58.591257 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:58.591257 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:58.591257 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:58.591257 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:58.591257 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:58.591257 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:58.591257 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:58.591257 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:58.591257 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 04:59:58.591257 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 04:59:58.591257 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 04:59:58.591257 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 04:59:58.592230 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:58.592230 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:58.592230 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 04:59:58.592230 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:59:58.592230 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 04:59:58.592230 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:59:58.592230 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:58.592230 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:58.592230 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:58.592230 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:58.592230 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:58.592230 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:58.592230 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:58.592230 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:58.592230 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:58.592230 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:58.592230 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:58.592230 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:58.593157 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:58.593157 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:58.593157 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 04:59:58.593157 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 04:59:58.593157 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 04:59:58.593157 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 04:59:58.593157 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 04:59:58.593157 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 04:59:58.593157 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 30 04:59:58.593157 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 04:59:58.593157 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 372/737] 10.222.0.77 () {56 vars in 1117 bytes} [Tue Aug 30 04:59:58 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:58.593157 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-477ea08f-e895-4749-be41-aebd922072d6 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:58.647458 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d32e2112-7138-4c43-a18c-cacab94e2fc0 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:58.685355 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ed5fff8-7124-483a-93bc-0878bf87911b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:58.685954 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ed5fff8-7124-483a-93bc-0878bf87911b None admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:58.686238 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ed5fff8-7124-483a-93bc-0878bf87911b None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:58.686507 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ed5fff8-7124-483a-93bc-0878bf87911b None admin] PATH_INFO: `/v3/services` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:58.687181 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ed5fff8-7124-483a-93bc-0878bf87911b None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:58.688559 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ed5fff8-7124-483a-93bc-0878bf87911b None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:58.694430 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:58.708473 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-477ea08f-e895-4749-be41-aebd922072d6 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:58.711569 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 366/738] 10.222.0.77 () {60 vars in 1141 bytes} [Tue Aug 30 04:59:58 2022] POST /identity/v3/services => generated 251 bytes in 184 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:59:58.712254 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4646996a-bc4e-4bc8-b061-3b6c7505edd1 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:58.712550 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4646996a-bc4e-4bc8-b061-3b6c7505edd1 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:58.712809 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4646996a-bc4e-4bc8-b061-3b6c7505edd1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:58.742437 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-4646996a-bc4e-4bc8-b061-3b6c7505edd1 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:59:58.744655 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-4646996a-bc4e-4bc8-b061-3b6c7505edd1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:59:58.764128 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:58.772040 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee392ef0-4442-4298-851f-5b283e50eded None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:58.773112 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee392ef0-4442-4298-851f-5b283e50eded None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:58.773643 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee392ef0-4442-4298-851f-5b283e50eded None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:58.774132 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee392ef0-4442-4298-851f-5b283e50eded None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:58.838307 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 367/739] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 04:59:58 2022] POST /identity/v3/auth/tokens => generated 903 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:59:58.839293 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4646996a-bc4e-4bc8-b061-3b6c7505edd1 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:58.865075 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee392ef0-4442-4298-851f-5b283e50eded None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:58.867158 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee392ef0-4442-4298-851f-5b283e50eded None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:58.963415 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4646996a-bc4e-4bc8-b061-3b6c7505edd1 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:58.995348 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:59.008049 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-526b7430-eaed-4f71-bfc4-9691ee1b3d07 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:59.009124 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 373/740] 10.222.0.77 () {58 vars in 1282 bytes} [Tue Aug 30 04:59:58 2022] GET /identity/v3/auth/tokens => generated 903 bytes in 416 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 04:59:59.009299 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-526b7430-eaed-4f71-bfc4-9691ee1b3d07 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:59.009703 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-526b7430-eaed-4f71-bfc4-9691ee1b3d07 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:59.010197 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-526b7430-eaed-4f71-bfc4-9691ee1b3d07 None admin] PATH_INFO: `/v3/users` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:59.011112 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-526b7430-eaed-4f71-bfc4-9691ee1b3d07 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:59.011557 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-526b7430-eaed-4f71-bfc4-9691ee1b3d07 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:59.013122 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-526b7430-eaed-4f71-bfc4-9691ee1b3d07 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:59.015019 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee392ef0-4442-4298-851f-5b283e50eded None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:59.036478 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 368/741] 10.222.0.77 () {56 vars in 1149 bytes} [Tue Aug 30 04:59:58 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:59.042920 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-526b7430-eaed-4f71-bfc4-9691ee1b3d07 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:59.057070 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:59.166211 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-526b7430-eaed-4f71-bfc4-9691ee1b3d07 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:59.174729 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee392ef0-4442-4298-851f-5b283e50eded None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:59.203082 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-585d4013-9fe0-4c70-81ec-2e45f75da6c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:59.204067 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-585d4013-9fe0-4c70-81ec-2e45f75da6c8 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:59.204615 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-585d4013-9fe0-4c70-81ec-2e45f75da6c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:59.204891 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-585d4013-9fe0-4c70-81ec-2e45f75da6c8 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:59.222598 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35848690-2fdc-4efe-a784-4df2dfa23527 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:59.223564 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35848690-2fdc-4efe-a784-4df2dfa23527 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:59.224070 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35848690-2fdc-4efe-a784-4df2dfa23527 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:59.224597 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35848690-2fdc-4efe-a784-4df2dfa23527 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:59.236040 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35848690-2fdc-4efe-a784-4df2dfa23527 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:59.237905 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35848690-2fdc-4efe-a784-4df2dfa23527 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:59.252639 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-35848690-2fdc-4efe-a784-4df2dfa23527 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:59:59.252639 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:59.252639 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:59.252639 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:59.252639 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:59.252639 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:59.252639 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:59.252639 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:59.252639 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:59.252639 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:59.252639 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:59.252639 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:59.252639 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:59:59.252639 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:59:59.252639 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:59:59.252639 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:59:59.253650 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:59.253650 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:59.253650 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:59.253650 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:59.253650 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:59.253650 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:59.253650 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:59.253650 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:59.253650 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:59.253650 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:59.253650 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:59.253650 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:59.253650 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:59.253650 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:59.253650 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:59.253650 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:59.253650 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:59:59.254561 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:59:59.254561 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:59:59.254561 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 04:59:59.261644 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 374/742] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 04:59:59 2022] GET /identity/v3/domains/Default => generated 87 bytes in 242 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:59.262755 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35848690-2fdc-4efe-a784-4df2dfa23527 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:59.267306 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:59.278343 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-585d4013-9fe0-4c70-81ec-2e45f75da6c8 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:59.279960 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-585d4013-9fe0-4c70-81ec-2e45f75da6c8 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:59.372995 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:59.394034 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35848690-2fdc-4efe-a784-4df2dfa23527 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:59.402540 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 369/743] 10.222.0.77 () {58 vars in 1282 bytes} [Tue Aug 30 04:59:59 2022] GET /identity/v3/auth/tokens => generated 903 bytes in 360 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 04:59:59.403566 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-585d4013-9fe0-4c70-81ec-2e45f75da6c8 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:59.430441 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27af068b-fda7-407c-91ae-db7ff8c607bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:59.431062 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27af068b-fda7-407c-91ae-db7ff8c607bb None admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:59.431340 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27af068b-fda7-407c-91ae-db7ff8c607bb None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:59.431607 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27af068b-fda7-407c-91ae-db7ff8c607bb None admin] PATH_INFO: `/v3/services` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:59.432250 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27af068b-fda7-407c-91ae-db7ff8c607bb None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:59.433611 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27af068b-fda7-407c-91ae-db7ff8c607bb None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:59.451950 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 375/744] 10.222.0.77 () {60 vars in 1141 bytes} [Tue Aug 30 04:59:59 2022] POST /identity/v3/services => generated 240 bytes in 189 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:59:59.452956 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27af068b-fda7-407c-91ae-db7ff8c607bb None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:59.497068 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:59.528548 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-585d4013-9fe0-4c70-81ec-2e45f75da6c8 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:59.576127 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27af068b-fda7-407c-91ae-db7ff8c607bb None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:59.603875 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7db4b093-6529-457c-a668-e39dab088a60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:59.604917 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7db4b093-6529-457c-a668-e39dab088a60 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:59.605416 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7db4b093-6529-457c-a668-e39dab088a60 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:59.605904 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7db4b093-6529-457c-a668-e39dab088a60 None admin] PATH_INFO: `/v3/domains` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:59.606759 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7db4b093-6529-457c-a668-e39dab088a60 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:59.607283 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7db4b093-6529-457c-a668-e39dab088a60 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:59.608882 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7db4b093-6529-457c-a668-e39dab088a60 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:59.613398 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-209008dd-0125-4460-a688-a71e07e59fe2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:59.614077 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-209008dd-0125-4460-a688-a71e07e59fe2 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:59.614355 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-209008dd-0125-4460-a688-a71e07e59fe2 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:59.614622 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-209008dd-0125-4460-a688-a71e07e59fe2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:59.624776 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 370/745] 10.222.0.77 () {56 vars in 1109 bytes} [Tue Aug 30 04:59:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:59.624987 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:59.634518 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-209008dd-0125-4460-a688-a71e07e59fe2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:59.636454 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7db4b093-6529-457c-a668-e39dab088a60 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:59.637936 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-209008dd-0125-4460-a688-a71e07e59fe2 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:59.667729 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-209008dd-0125-4460-a688-a71e07e59fe2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:59:59.667729 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:59.667729 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:59.667729 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:59.667729 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:59.667729 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:59.667729 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:59.667729 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:59.667729 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:59.667729 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:59.667729 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:59.667729 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:59.667729 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:59:59.667729 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:59:59.667729 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:59:59.667729 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:59:59.668836 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:59.668836 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:59.668836 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:59.668836 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:59.668836 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:59.668836 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:59.668836 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:59.668836 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:59.668836 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:59.668836 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:59.668836 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:59.668836 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:59.668836 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:59.668836 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:59.668836 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:59.668836 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:59.668836 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:59:59.669717 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:59:59.669717 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:59:59.669717 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 04:59:59.669717 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 376/746] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 04:59:59 2022] GET /identity/v3/domains/Default => generated 87 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:59.674339 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-209008dd-0125-4460-a688-a71e07e59fe2 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:59.800248 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7db4b093-6529-457c-a668-e39dab088a60 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:59.820300 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-209008dd-0125-4460-a688-a71e07e59fe2 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:59:59.843340 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba713f0f-bd27-4905-979e-5f7c342b8558 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:59.844004 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba713f0f-bd27-4905-979e-5f7c342b8558 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:59.844235 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba713f0f-bd27-4905-979e-5f7c342b8558 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:59.844500 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba713f0f-bd27-4905-979e-5f7c342b8558 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:59.853481 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba713f0f-bd27-4905-979e-5f7c342b8558 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:59.855466 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba713f0f-bd27-4905-979e-5f7c342b8558 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:59.862529 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cb23ed3-fe58-4866-9b20-97ba62b5b433 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:59:59.863451 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cb23ed3-fe58-4866-9b20-97ba62b5b433 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:59:59.863736 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cb23ed3-fe58-4866-9b20-97ba62b5b433 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:59:59.864009 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cb23ed3-fe58-4866-9b20-97ba62b5b433 None admin] PATH_INFO: `/v3/domains` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:59:59.864656 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cb23ed3-fe58-4866-9b20-97ba62b5b433 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:59:59.864955 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cb23ed3-fe58-4866-9b20-97ba62b5b433 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:59:59.866893 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cb23ed3-fe58-4866-9b20-97ba62b5b433 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:59:59.868820 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-ba713f0f-bd27-4905-979e-5f7c342b8558 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:59:59.868820 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:59:59.868820 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:59:59.868820 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:59:59.868820 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:59:59.868820 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:59:59.868820 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:59:59.868820 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:59:59.868820 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:59:59.868820 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:59:59.868820 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:59:59.868820 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:59:59.868820 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 04:59:59.868820 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 04:59:59.868820 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 04:59:59.870788 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 04:59:59.870788 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:59:59.870788 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:59:59.870788 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:59:59.870788 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:59:59.870788 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:59:59.870788 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:59:59.870788 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:59:59.870788 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 04:59:59.870788 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:59:59.870788 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:59:59.870788 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:59:59.870788 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:59:59.870788 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:59:59.870788 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:59:59.870788 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:59:59.870788 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:59:59.872170 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 04:59:59.872170 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 04:59:59.872170 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 04:59:59.872170 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 04:59:59.872170 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 04:59:59.872170 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 04:59:59.872170 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:59:59.872170 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 04:59:59.872170 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 371/747] 10.222.0.77 () {56 vars in 1111 bytes} [Tue Aug 30 04:59:59 2022] GET /identity/v3/projects/service => generated 88 bytes in 238 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:59:59.878315 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba713f0f-bd27-4905-979e-5f7c342b8558 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:59.883061 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 377/748] 10.222.0.77 () {56 vars in 1109 bytes} [Tue Aug 30 04:59:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:59:59.884457 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:59:59.888014 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cb23ed3-fe58-4866-9b20-97ba62b5b433 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:59:59.959878 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:00.034856 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cb23ed3-fe58-4866-9b20-97ba62b5b433 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:00.059227 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba713f0f-bd27-4905-979e-5f7c342b8558 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:00.100270 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9419ad6d-bbb6-424e-ab37-06f3e1e6ce3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:00.101443 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9419ad6d-bbb6-424e-ab37-06f3e1e6ce3a None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:00.101725 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9419ad6d-bbb6-424e-ab37-06f3e1e6ce3a None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:00.101995 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9419ad6d-bbb6-424e-ab37-06f3e1e6ce3a None admin] PATH_INFO: `/v3/projects/service` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:00.114122 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9419ad6d-bbb6-424e-ab37-06f3e1e6ce3a None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:00.115607 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9419ad6d-bbb6-424e-ab37-06f3e1e6ce3a None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:00.129780 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56949a10-07bd-430a-89e1-53cb58546b07 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:00.130395 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56949a10-07bd-430a-89e1-53cb58546b07 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:00.130676 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56949a10-07bd-430a-89e1-53cb58546b07 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:00.130980 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56949a10-07bd-430a-89e1-53cb58546b07 None admin] PATH_INFO: `/v3/projects` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:00.131681 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56949a10-07bd-430a-89e1-53cb58546b07 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:00.132021 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56949a10-07bd-430a-89e1-53cb58546b07 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:00.138103 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-9419ad6d-bbb6-424e-ab37-06f3e1e6ce3a None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 05:00:00.138103 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:00.138103 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:00:00.138103 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:00:00.138103 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:00:00.138103 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:00:00.138103 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:00:00.138103 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:00:00.138103 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:00:00.138103 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:00:00.138103 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:00:00.138103 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:00:00.138103 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 05:00:00.138103 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 05:00:00.138103 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 05:00:00.140101 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 05:00:00.140101 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:00:00.140101 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:00:00.140101 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:00:00.140101 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:00:00.140101 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:00:00.140101 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:00:00.140101 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:00:00.140101 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 05:00:00.140101 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:00:00.140101 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:00:00.140101 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:00:00.140101 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:00:00.140101 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:00:00.140101 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:00:00.140101 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:00:00.140101 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:00:00.142208 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 05:00:00.142208 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 05:00:00.142208 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 05:00:00.142208 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 05:00:00.142208 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 05:00:00.142208 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 05:00:00.142208 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 05:00:00.142208 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:00:00.143258 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56949a10-07bd-430a-89e1-53cb58546b07 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:00.146538 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 378/749] 10.222.0.77 () {56 vars in 1111 bytes} [Tue Aug 30 04:59:59 2022] GET /identity/v3/projects/service => generated 88 bytes in 259 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:00:00.148728 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9419ad6d-bbb6-424e-ab37-06f3e1e6ce3a None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:00.160952 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 372/750] 10.222.0.77 () {56 vars in 1148 bytes} [Tue Aug 30 04:59:59 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 283 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:00.167472 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56949a10-07bd-430a-89e1-53cb58546b07 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:00.274071 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:00.300535 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56949a10-07bd-430a-89e1-53cb58546b07 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:00.339113 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b99d42a6-2e83-4b22-9f26-4239501904b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:00.340154 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b99d42a6-2e83-4b22-9f26-4239501904b4 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:00.340430 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b99d42a6-2e83-4b22-9f26-4239501904b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:00.340697 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b99d42a6-2e83-4b22-9f26-4239501904b4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:00.341416 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b99d42a6-2e83-4b22-9f26-4239501904b4 None admin] RBAC: Adding query filter params (role.id=9d9d5d2c270146488249b7ed943280db, scope.project.id=e9919141e3b947ab8d91b55bba4a7de6, user.id=e0a7f7afc3d44ec788c05d005e52a1b8) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:00.341715 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b99d42a6-2e83-4b22-9f26-4239501904b4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:00.343159 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b99d42a6-2e83-4b22-9f26-4239501904b4 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:00.349608 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 373/751] 10.222.0.77 () {56 vars in 1374 bytes} [Tue Aug 30 05:00:00 2022] GET /identity/v3/role_assignments?user.id=e0a7f7afc3d44ec788c05d005e52a1b8&scope.project.id=e9919141e3b947ab8d91b55bba4a7de6&role.id=9d9d5d2c270146488249b7ed943280db => generated 259 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 05:00:00.366310 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9419ad6d-bbb6-424e-ab37-06f3e1e6ce3a None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:00.381479 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:00.405112 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7875c597-09ac-4ef3-bc78-56b005bd8d02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:00.405725 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7875c597-09ac-4ef3-bc78-56b005bd8d02 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:00.406001 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7875c597-09ac-4ef3-bc78-56b005bd8d02 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:00.406269 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7875c597-09ac-4ef3-bc78-56b005bd8d02 None admin] PATH_INFO: `/v3/projects` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:00.406985 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7875c597-09ac-4ef3-bc78-56b005bd8d02 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:00.407285 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7875c597-09ac-4ef3-bc78-56b005bd8d02 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:00.408713 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7875c597-09ac-4ef3-bc78-56b005bd8d02 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:00.422504 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 379/752] 10.222.0.77 () {56 vars in 1148 bytes} [Tue Aug 30 05:00:00 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 274 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:00.428128 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b99d42a6-2e83-4b22-9f26-4239501904b4 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:00.499926 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba1e1921-3bee-458e-ad43-edbd251e8e5d None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:00.500240 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba1e1921-3bee-458e-ad43-edbd251e8e5d None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:00.500461 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba1e1921-3bee-458e-ad43-edbd251e8e5d None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:00.502177 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 380/753] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:00:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:00:00.507775 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac2f86b2-b044-4f1a-8e81-64900a118072 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:00.508036 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac2f86b2-b044-4f1a-8e81-64900a118072 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:00.508309 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac2f86b2-b044-4f1a-8e81-64900a118072 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:00.527946 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-ac2f86b2-b044-4f1a-8e81-64900a118072 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:00:00.529158 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-ac2f86b2-b044-4f1a-8e81-64900a118072 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:00:00.602054 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 381/754] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:00:00 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 30 05:00:00.611233 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b99d42a6-2e83-4b22-9f26-4239501904b4 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:00.656710 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:00.665270 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da46cc6d-9dc8-4bb6-a12d-654548648333 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:00.666317 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da46cc6d-9dc8-4bb6-a12d-654548648333 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:00.666595 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da46cc6d-9dc8-4bb6-a12d-654548648333 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:00.666896 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da46cc6d-9dc8-4bb6-a12d-654548648333 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:00.667600 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da46cc6d-9dc8-4bb6-a12d-654548648333 None admin] RBAC: Adding query filter params (role.id=821647e8b40542499667e29e1b301cb2, scope.project.id=e9919141e3b947ab8d91b55bba4a7de6, user.id=fe51dbec91624ed1ab00b441e29a3dd5) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:00.667903 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da46cc6d-9dc8-4bb6-a12d-654548648333 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:00.669347 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da46cc6d-9dc8-4bb6-a12d-654548648333 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:00.679723 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 374/755] 10.222.0.77 () {56 vars in 1374 bytes} [Tue Aug 30 05:00:00 2022] GET /identity/v3/role_assignments?user.id=fe51dbec91624ed1ab00b441e29a3dd5&scope.project.id=e9919141e3b947ab8d91b55bba4a7de6&role.id=821647e8b40542499667e29e1b301cb2 => generated 259 bytes in 252 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:00.928143 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac2f86b2-b044-4f1a-8e81-64900a118072 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:01.037826 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac2f86b2-b044-4f1a-8e81-64900a118072 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:01.039274 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-106ac2c8-ec76-448f-850b-f0adc2251458 None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:01.039730 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-106ac2c8-ec76-448f-850b-f0adc2251458 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:01.039810 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-106ac2c8-ec76-448f-850b-f0adc2251458 None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:01.039989 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:01.041368 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 375/756] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:00:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:00:01.046420 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d8da9ef-f010-4de6-9142-e40082b12313 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:01.046698 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d8da9ef-f010-4de6-9142-e40082b12313 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:01.047099 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d8da9ef-f010-4de6-9142-e40082b12313 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:01.079112 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-4d8da9ef-f010-4de6-9142-e40082b12313 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:00:01.080372 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-4d8da9ef-f010-4de6-9142-e40082b12313 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:00:01.107692 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afa4e74f-3d19-4001-ae6d-3b5631d35538 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:01.108326 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afa4e74f-3d19-4001-ae6d-3b5631d35538 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:01.108601 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afa4e74f-3d19-4001-ae6d-3b5631d35538 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:01.108863 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afa4e74f-3d19-4001-ae6d-3b5631d35538 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:01.121664 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afa4e74f-3d19-4001-ae6d-3b5631d35538 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 05:00:01.126670 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-afa4e74f-3d19-4001-ae6d-3b5631d35538 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 05:00:01.126670 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:01.126670 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:00:01.126670 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:00:01.126670 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:00:01.126670 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:00:01.126670 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:00:01.126670 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:00:01.126670 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:00:01.126670 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:00:01.126670 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:00:01.126670 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:00:01.126670 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 05:00:01.126670 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 05:00:01.126670 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 05:00:01.126670 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 05:00:01.127714 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 05:00:01.127714 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 05:00:01.127714 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 05:00:01.127714 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 05:00:01.127714 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:00:01.127714 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:00:01.127714 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:00:01.127714 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:00:01.127714 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:00:01.127714 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:00:01.127714 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:00:01.127714 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 05:00:01.127714 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:00:01.127714 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:00:01.127714 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:00:01.127714 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:00:01.128556 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:00:01.128556 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:00:01.128556 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:00:01.128556 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:00:01.128556 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 05:00:01.128556 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 05:00:01.128556 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 05:00:01.128556 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 05:00:01.128556 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 05:00:01.128556 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 05:00:01.128556 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 05:00:01.128556 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:00:01.128556 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 382/757] 10.222.0.77 () {56 vars in 1096 bytes} [Tue Aug 30 05:00:00 2022] GET /identity/v3/roles/admin => generated 83 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:00:01.133542 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afa4e74f-3d19-4001-ae6d-3b5631d35538 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:01.227462 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 376/758] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:00:01 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 05:00:01.243109 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afa4e74f-3d19-4001-ae6d-3b5631d35538 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:01.277022 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78667bb7-b098-4a0e-8943-9be66bea689d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:01.277690 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78667bb7-b098-4a0e-8943-9be66bea689d None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:01.277965 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78667bb7-b098-4a0e-8943-9be66bea689d None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:01.278230 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78667bb7-b098-4a0e-8943-9be66bea689d None admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:01.278906 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78667bb7-b098-4a0e-8943-9be66bea689d None admin] RBAC: Adding query filter params (name=admin) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:01.279202 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78667bb7-b098-4a0e-8943-9be66bea689d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:01.280555 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78667bb7-b098-4a0e-8943-9be66bea689d None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:01.294063 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 383/759] 10.222.0.77 () {56 vars in 1099 bytes} [Tue Aug 30 05:00:01 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:01.295477 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d8da9ef-f010-4de6-9142-e40082b12313 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:01.400242 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d8da9ef-f010-4de6-9142-e40082b12313 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:01.424626 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:01.436564 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6148662-7506-4b5b-8e69-10e0676fd402 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:01.437189 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6148662-7506-4b5b-8e69-10e0676fd402 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:01.437465 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6148662-7506-4b5b-8e69-10e0676fd402 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:01.437736 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6148662-7506-4b5b-8e69-10e0676fd402 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:01.448392 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6148662-7506-4b5b-8e69-10e0676fd402 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:01.449905 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6148662-7506-4b5b-8e69-10e0676fd402 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:01.465942 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-e6148662-7506-4b5b-8e69-10e0676fd402 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 05:00:01.465942 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:01.465942 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:00:01.465942 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:00:01.465942 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:00:01.465942 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:00:01.465942 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:00:01.465942 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:00:01.465942 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:00:01.465942 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:00:01.465942 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:00:01.465942 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:00:01.465942 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 05:00:01.465942 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 05:00:01.465942 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 05:00:01.465942 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 05:00:01.467048 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:00:01.467048 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:00:01.467048 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:00:01.467048 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:00:01.467048 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:00:01.467048 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:00:01.467048 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:00:01.467048 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 05:00:01.467048 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:00:01.467048 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:00:01.467048 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:00:01.467048 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:00:01.467048 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:00:01.467048 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:00:01.467048 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:00:01.467048 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:00:01.467048 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 05:00:01.468032 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 05:00:01.468032 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 05:00:01.468032 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:00:01.471193 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 377/760] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 05:00:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:00:01.472871 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78667bb7-b098-4a0e-8943-9be66bea689d None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:01.519295 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:01.595744 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78667bb7-b098-4a0e-8943-9be66bea689d None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:01.639049 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e22fdc6-3da8-419d-8a53-606f948a9c0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:01.639674 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e22fdc6-3da8-419d-8a53-606f948a9c0e None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:01.639911 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e22fdc6-3da8-419d-8a53-606f948a9c0e None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:01.640180 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e22fdc6-3da8-419d-8a53-606f948a9c0e None admin] PATH_INFO: `/v3/domains` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:01.640811 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e22fdc6-3da8-419d-8a53-606f948a9c0e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:01.641165 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e22fdc6-3da8-419d-8a53-606f948a9c0e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:01.642565 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e22fdc6-3da8-419d-8a53-606f948a9c0e None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:01.658710 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 384/761] 10.222.0.77 () {56 vars in 1109 bytes} [Tue Aug 30 05:00:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:01.664305 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6148662-7506-4b5b-8e69-10e0676fd402 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:01.685699 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e22fdc6-3da8-419d-8a53-606f948a9c0e None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:01.791025 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6148662-7506-4b5b-8e69-10e0676fd402 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:01.828955 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15fe01dc-a044-4b1b-adfd-cdca8fbbc321 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:01.829585 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15fe01dc-a044-4b1b-adfd-cdca8fbbc321 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:01.829862 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15fe01dc-a044-4b1b-adfd-cdca8fbbc321 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:01.830183 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15fe01dc-a044-4b1b-adfd-cdca8fbbc321 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:01.838540 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15fe01dc-a044-4b1b-adfd-cdca8fbbc321 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:01.840636 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15fe01dc-a044-4b1b-adfd-cdca8fbbc321 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:01.853768 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-15fe01dc-a044-4b1b-adfd-cdca8fbbc321 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 30 05:00:01.853768 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:01.853768 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:00:01.853768 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:00:01.853768 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:00:01.853768 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:00:01.853768 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:00:01.853768 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:00:01.853768 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:00:01.853768 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:00:01.853768 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:00:01.853768 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:00:01.853768 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 05:00:01.853768 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 05:00:01.853768 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 05:00:01.853768 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 05:00:01.854696 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:00:01.854696 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:00:01.854696 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 05:00:01.854696 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 05:00:01.854696 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 05:00:01.854696 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 05:00:01.854696 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:00:01.854696 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:00:01.854696 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:00:01.854696 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:00:01.854696 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:00:01.854696 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 05:00:01.854696 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:00:01.854696 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:00:01.854696 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:00:01.854696 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:00:01.854696 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:00:01.854696 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:00:01.855652 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:00:01.855652 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:00:01.855652 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 05:00:01.855652 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 05:00:01.855652 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 05:00:01.855652 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 05:00:01.855652 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 05:00:01.855652 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 05:00:01.855652 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 30 05:00:01.855652 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:00:01.858954 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 378/762] 10.222.0.77 () {56 vars in 1093 bytes} [Tue Aug 30 05:00:01 2022] GET /identity/v3/users/nova => generated 82 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:00:01.861753 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15fe01dc-a044-4b1b-adfd-cdca8fbbc321 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:01.889012 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:01.905459 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:01.948750 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e22fdc6-3da8-419d-8a53-606f948a9c0e None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:01.978983 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15fe01dc-a044-4b1b-adfd-cdca8fbbc321 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:01.995639 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25c68ce9-eb63-4414-b865-e5e0d6f85e13 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:01.996366 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25c68ce9-eb63-4414-b865-e5e0d6f85e13 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:01.996673 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25c68ce9-eb63-4414-b865-e5e0d6f85e13 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:01.996943 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25c68ce9-eb63-4414-b865-e5e0d6f85e13 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:01.997517 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25c68ce9-eb63-4414-b865-e5e0d6f85e13 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:01.998909 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25c68ce9-eb63-4414-b865-e5e0d6f85e13 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:02.009252 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-25c68ce9-eb63-4414-b865-e5e0d6f85e13 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 30 05:00:02.009252 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:02.009252 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 30 05:00:02.009252 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 30 05:00:02.009252 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 30 05:00:02.009252 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 30 05:00:02.009252 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 30 05:00:02.009252 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 05:00:02.009252 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 30 05:00:02.009252 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:00:02.009252 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 05:00:02.009252 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:00:02.009252 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:02.009252 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:00:02.009252 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:00:02.009252 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:00:02.009252 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:00:02.010223 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:00:02.010223 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:00:02.010223 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:00:02.010223 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:00:02.010223 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:00:02.010223 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:00:02.010223 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 30 05:00:02.010223 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 30 05:00:02.010223 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 30 05:00:02.010223 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 30 05:00:02.010223 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:00:02.010223 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:00:02.010223 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:00:02.010223 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:00:02.010223 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:00:02.010223 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:00:02.011089 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:00:02.011089 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 05:00:02.011089 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:00:02.011089 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:00:02.011089 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:00:02.011089 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:00:02.011089 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:00:02.011089 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:00:02.011089 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:00:02.011089 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:00:02.011089 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 30 05:00:02.011089 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 05:00:02.011089 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 30 05:00:02.011089 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:00:02.011089 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 385/763] 10.222.0.77 () {56 vars in 1129 bytes} [Tue Aug 30 05:00:01 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 325 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 05:00:02.011870 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3757269b-9435-45d9-a79e-87916522cab7 None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:02.012464 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3757269b-9435-45d9-a79e-87916522cab7 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:02.012464 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3757269b-9435-45d9-a79e-87916522cab7 None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:02.013643 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 386/764] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:00:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:00:02.017525 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3757269b-9435-45d9-a79e-87916522cab7 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:02.026643 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39191024-0700-487e-8519-799ad1d2e99f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:02.027311 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39191024-0700-487e-8519-799ad1d2e99f None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:02.028088 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39191024-0700-487e-8519-799ad1d2e99f None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:02.028369 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39191024-0700-487e-8519-799ad1d2e99f None admin] PATH_INFO: `/v3/users` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:02.029105 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39191024-0700-487e-8519-799ad1d2e99f None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:02.029413 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39191024-0700-487e-8519-799ad1d2e99f None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:02.031781 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39191024-0700-487e-8519-799ad1d2e99f None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:02.055108 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 379/765] 10.222.0.77 () {56 vars in 1133 bytes} [Tue Aug 30 05:00:01 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:02.056360 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6035ee72-675f-4d2f-b455-ef4f7621fbe7 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:02.056648 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6035ee72-675f-4d2f-b455-ef4f7621fbe7 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:02.056921 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6035ee72-675f-4d2f-b455-ef4f7621fbe7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:02.078779 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-6035ee72-675f-4d2f-b455-ef4f7621fbe7 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:00:02.080021 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-6035ee72-675f-4d2f-b455-ef4f7621fbe7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:00:02.138608 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3757269b-9435-45d9-a79e-87916522cab7 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:02.166912 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 380/766] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:00:02 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 05:00:02.167853 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6035ee72-675f-4d2f-b455-ef4f7621fbe7 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:02.191334 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b65d136-daa0-4274-a7c1-a4f63f84abde None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:02.191945 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b65d136-daa0-4274-a7c1-a4f63f84abde None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:02.192349 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b65d136-daa0-4274-a7c1-a4f63f84abde None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:02.192632 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b65d136-daa0-4274-a7c1-a4f63f84abde None admin] PATH_INFO: `/v3/services` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:02.194435 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b65d136-daa0-4274-a7c1-a4f63f84abde None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:02.198875 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b65d136-daa0-4274-a7c1-a4f63f84abde None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:02.198875 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b65d136-daa0-4274-a7c1-a4f63f84abde None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:02.208279 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 387/767] 10.222.0.77 () {56 vars in 1124 bytes} [Tue Aug 30 05:00:02 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 05:00:02.216052 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b65d136-daa0-4274-a7c1-a4f63f84abde None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:02.223866 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:02.246062 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:02.321685 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6035ee72-675f-4d2f-b455-ef4f7621fbe7 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:02.337558 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b65d136-daa0-4274-a7c1-a4f63f84abde None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:02.357631 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc84d65a-f8c2-401c-a0d9-712b198ddd37 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:02.358397 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc84d65a-f8c2-401c-a0d9-712b198ddd37 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:02.358769 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc84d65a-f8c2-401c-a0d9-712b198ddd37 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:02.359230 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc84d65a-f8c2-401c-a0d9-712b198ddd37 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:02.370128 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc84d65a-f8c2-401c-a0d9-712b198ddd37 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:02.371789 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc84d65a-f8c2-401c-a0d9-712b198ddd37 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:02.384859 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-847b4382-935e-405a-8928-c8a3d9045334 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:02.385677 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-847b4382-935e-405a-8928-c8a3d9045334 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:02.386102 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-847b4382-935e-405a-8928-c8a3d9045334 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:02.386550 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-847b4382-935e-405a-8928-c8a3d9045334 None admin] PATH_INFO: `/v3/services` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:02.387397 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-847b4382-935e-405a-8928-c8a3d9045334 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:02.387512 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-fc84d65a-f8c2-401c-a0d9-712b198ddd37 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 05:00:02.387512 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:02.387512 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:00:02.387512 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:00:02.387512 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:00:02.387512 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:00:02.387512 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:00:02.387512 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:00:02.387512 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:00:02.387512 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:00:02.387512 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:00:02.387512 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:00:02.387512 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 05:00:02.387512 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 05:00:02.387512 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 05:00:02.387512 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 05:00:02.388658 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:00:02.388658 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:00:02.388658 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:00:02.388658 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:00:02.388658 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:00:02.388658 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:00:02.388658 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:00:02.388658 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 05:00:02.388658 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:00:02.388658 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:00:02.388658 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:00:02.388658 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:00:02.388658 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:00:02.388658 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:00:02.388658 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:00:02.388658 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:00:02.388658 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 05:00:02.389759 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 05:00:02.389759 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 05:00:02.389759 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:00:02.390355 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-847b4382-935e-405a-8928-c8a3d9045334 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:02.392004 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-847b4382-935e-405a-8928-c8a3d9045334 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:02.399727 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 381/768] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 05:00:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 224 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:00:02.399727 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc84d65a-f8c2-401c-a0d9-712b198ddd37 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:02.408712 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 388/769] 10.222.0.77 () {56 vars in 1124 bytes} [Tue Aug 30 05:00:02 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:02.419772 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-847b4382-935e-405a-8928-c8a3d9045334 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:02.477684 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:02.505803 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc84d65a-f8c2-401c-a0d9-712b198ddd37 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:02.541660 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12e84429-9300-4df8-8127-d17a6c001b4c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:02.542513 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e84429-9300-4df8-8127-d17a6c001b4c None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:02.542992 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e84429-9300-4df8-8127-d17a6c001b4c None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:02.543406 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e84429-9300-4df8-8127-d17a6c001b4c None admin] PATH_INFO: `/v3/domains` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:02.544186 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12e84429-9300-4df8-8127-d17a6c001b4c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:02.544629 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12e84429-9300-4df8-8127-d17a6c001b4c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:02.546152 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12e84429-9300-4df8-8127-d17a6c001b4c None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:02.563716 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 382/770] 10.222.0.77 () {56 vars in 1109 bytes} [Tue Aug 30 05:00:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 05:00:02.565150 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12e84429-9300-4df8-8127-d17a6c001b4c None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:02.578556 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:02.618363 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-847b4382-935e-405a-8928-c8a3d9045334 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:02.671297 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc551065-5384-4956-bf77-c2ac45872c68 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:02.671967 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc551065-5384-4956-bf77-c2ac45872c68 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:02.672610 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc551065-5384-4956-bf77-c2ac45872c68 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:02.672892 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc551065-5384-4956-bf77-c2ac45872c68 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:02.673875 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc551065-5384-4956-bf77-c2ac45872c68 None admin] RBAC: Adding query filter params (interface=public, service_id=5e97646e5e884b21949bd6e5bae8e845, region_id=RegionOne) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:02.674185 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc551065-5384-4956-bf77-c2ac45872c68 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:02.675578 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc551065-5384-4956-bf77-c2ac45872c68 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:02.675897 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12e84429-9300-4df8-8127-d17a6c001b4c None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:02.691192 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 389/771] 10.222.0.77 () {56 vars in 1251 bytes} [Tue Aug 30 05:00:02 2022] GET /identity/v3/endpoints?service_id=5e97646e5e884b21949bd6e5bae8e845&interface=public®ion_id=RegionOne => generated 194 bytes in 272 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:02.692089 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc551065-5384-4956-bf77-c2ac45872c68 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:02.718503 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe3dfc77-6f05-42cc-b229-3fa3a506ad8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:02.719501 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe3dfc77-6f05-42cc-b229-3fa3a506ad8d None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:02.720003 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe3dfc77-6f05-42cc-b229-3fa3a506ad8d None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:02.720478 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe3dfc77-6f05-42cc-b229-3fa3a506ad8d None admin] PATH_INFO: `/v3/projects/service` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:02.723154 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:02.731073 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe3dfc77-6f05-42cc-b229-3fa3a506ad8d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:02.732770 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe3dfc77-6f05-42cc-b229-3fa3a506ad8d None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:02.744628 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-fe3dfc77-6f05-42cc-b229-3fa3a506ad8d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 05:00:02.744628 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:02.744628 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:00:02.744628 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:00:02.744628 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:00:02.744628 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:00:02.744628 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:00:02.744628 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:00:02.744628 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:00:02.744628 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:00:02.744628 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:00:02.744628 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:00:02.744628 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 05:00:02.744628 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 05:00:02.744628 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 05:00:02.745515 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 05:00:02.745515 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:00:02.745515 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:00:02.745515 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:00:02.745515 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:00:02.745515 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:00:02.745515 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:00:02.745515 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:00:02.745515 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 05:00:02.745515 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:00:02.745515 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:00:02.745515 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:00:02.745515 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:00:02.745515 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:00:02.745515 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:00:02.745515 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:00:02.745515 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:00:02.746390 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 05:00:02.746390 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 05:00:02.746390 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 05:00:02.746390 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 05:00:02.746390 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 05:00:02.746390 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 05:00:02.746390 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 05:00:02.746390 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:00:02.752897 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 383/772] 10.222.0.77 () {56 vars in 1111 bytes} [Tue Aug 30 05:00:02 2022] GET /identity/v3/projects/service => generated 88 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 05:00:02.753977 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe3dfc77-6f05-42cc-b229-3fa3a506ad8d None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:02.815272 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc551065-5384-4956-bf77-c2ac45872c68 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:02.852538 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdb033ba-92e2-4fd9-87a9-333e8df1e12c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:02.853392 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb033ba-92e2-4fd9-87a9-333e8df1e12c None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:02.853667 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb033ba-92e2-4fd9-87a9-333e8df1e12c None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:02.853941 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb033ba-92e2-4fd9-87a9-333e8df1e12c None admin] PATH_INFO: `/v3/services/network` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:02.854566 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdb033ba-92e2-4fd9-87a9-333e8df1e12c None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:02.855941 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdb033ba-92e2-4fd9-87a9-333e8df1e12c None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:02.864928 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-cdb033ba-92e2-4fd9-87a9-333e8df1e12c None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Aug 30 05:00:02.864928 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:02.864928 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 30 05:00:02.864928 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 30 05:00:02.864928 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 30 05:00:02.864928 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 30 05:00:02.864928 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 30 05:00:02.864928 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 05:00:02.864928 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Aug 30 05:00:02.864928 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:00:02.864928 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 05:00:02.864928 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:00:02.864928 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:02.864928 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:00:02.864928 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:00:02.864928 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:00:02.864928 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:00:02.865897 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:00:02.865897 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:00:02.865897 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:00:02.865897 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:00:02.865897 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:00:02.865897 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:00:02.865897 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 30 05:00:02.865897 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 30 05:00:02.865897 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 30 05:00:02.865897 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 30 05:00:02.865897 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:00:02.865897 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:00:02.865897 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:00:02.865897 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:00:02.865897 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:00:02.865897 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:00:02.866763 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:00:02.866763 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 05:00:02.866763 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:00:02.866763 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:00:02.866763 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:00:02.866763 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:00:02.866763 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:00:02.866763 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:00:02.866763 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:00:02.866763 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:00:02.866763 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 30 05:00:02.866763 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 05:00:02.866763 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Aug 30 05:00:02.866763 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:00:02.866763 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 390/773] 10.222.0.77 () {56 vars in 1111 bytes} [Tue Aug 30 05:00:02 2022] GET /identity/v3/services/network => generated 88 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:00:02.867739 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdb033ba-92e2-4fd9-87a9-333e8df1e12c None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:02.875501 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:02.887829 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe3dfc77-6f05-42cc-b229-3fa3a506ad8d None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:02.926408 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e51c3dd-5c9f-48c4-9689-8076e6eae74c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:02.927271 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e51c3dd-5c9f-48c4-9689-8076e6eae74c None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:02.927691 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e51c3dd-5c9f-48c4-9689-8076e6eae74c None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:02.928129 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e51c3dd-5c9f-48c4-9689-8076e6eae74c None admin] PATH_INFO: `/v3/services` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:02.928886 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e51c3dd-5c9f-48c4-9689-8076e6eae74c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:02.930385 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e51c3dd-5c9f-48c4-9689-8076e6eae74c None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:02.936034 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 384/774] 10.222.0.77 () {56 vars in 1087 bytes} [Tue Aug 30 05:00:02 2022] GET /identity/v3/services => generated 772 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:02.937293 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a82c3fb6-559e-4843-ae70-f0ad4e8bc79e None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:02.937719 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a82c3fb6-559e-4843-ae70-f0ad4e8bc79e None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:02.938160 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a82c3fb6-559e-4843-ae70-f0ad4e8bc79e None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:02.939540 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 385/775] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:00:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:00:02.940404 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a82c3fb6-559e-4843-ae70-f0ad4e8bc79e None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:03.007502 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdb033ba-92e2-4fd9-87a9-333e8df1e12c None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:03.015831 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:03.055292 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c9ecadf-44f4-4903-8798-ff1cf86ea295 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:03.056248 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c9ecadf-44f4-4903-8798-ff1cf86ea295 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:03.056707 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c9ecadf-44f4-4903-8798-ff1cf86ea295 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:03.057152 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c9ecadf-44f4-4903-8798-ff1cf86ea295 None admin] PATH_INFO: `/v3/projects` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:03.058071 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c9ecadf-44f4-4903-8798-ff1cf86ea295 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:03.058596 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c9ecadf-44f4-4903-8798-ff1cf86ea295 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:03.060355 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c9ecadf-44f4-4903-8798-ff1cf86ea295 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:03.075291 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 391/776] 10.222.0.77 () {56 vars in 1148 bytes} [Tue Aug 30 05:00:02 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:03.076928 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaed2579-72f8-4797-8480-0ae22664ded1 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:03.077407 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaed2579-72f8-4797-8480-0ae22664ded1 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:03.077899 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaed2579-72f8-4797-8480-0ae22664ded1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:03.081183 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a82c3fb6-559e-4843-ae70-f0ad4e8bc79e None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:03.103943 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-eaed2579-72f8-4797-8480-0ae22664ded1 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:00:03.105451 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-eaed2579-72f8-4797-8480-0ae22664ded1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:00:03.126918 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f812392-0766-4fd4-9373-b8a98845cd8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:03.127612 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f812392-0766-4fd4-9373-b8a98845cd8e None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:03.127897 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f812392-0766-4fd4-9373-b8a98845cd8e None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:03.128159 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f812392-0766-4fd4-9373-b8a98845cd8e None admin] PATH_INFO: `/v3/services` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:03.129218 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f812392-0766-4fd4-9373-b8a98845cd8e None admin] RBAC: Adding query filter params (name=network) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:03.129515 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f812392-0766-4fd4-9373-b8a98845cd8e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:03.130915 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f812392-0766-4fd4-9373-b8a98845cd8e None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:03.136131 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 386/777] 10.222.0.77 () {56 vars in 1112 bytes} [Tue Aug 30 05:00:02 2022] GET /identity/v3/services?name=network => generated 124 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:03.136916 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f812392-0766-4fd4-9373-b8a98845cd8e None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:03.173719 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:03.182743 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 392/778] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:00:03 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 05:00:03.192148 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b348c646-7d76-4007-b4b2-2d9a3efc2d9a None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:03.193158 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b348c646-7d76-4007-b4b2-2d9a3efc2d9a None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:03.193619 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b348c646-7d76-4007-b4b2-2d9a3efc2d9a None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:03.196124 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 393/779] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:00:03 2022] GET /identity => generated 270 bytes in 13 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:00:03.197126 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b348c646-7d76-4007-b4b2-2d9a3efc2d9a None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:03.248979 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f812392-0766-4fd4-9373-b8a98845cd8e None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:03.274416 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:03.286497 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05ae507a-4ea9-4fa1-9d03-84dffe3af16f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:03.287336 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05ae507a-4ea9-4fa1-9d03-84dffe3af16f None admin] REQUEST_METHOD: `PUT` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:03.287619 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05ae507a-4ea9-4fa1-9d03-84dffe3af16f None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:03.287940 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05ae507a-4ea9-4fa1-9d03-84dffe3af16f None admin] PATH_INFO: `/v3/projects/e9919141e3b947ab8d91b55bba4a7de6/users/886eb5e00bbf4ec292eb962b665b37d0/roles/9d9d5d2c270146488249b7ed943280db` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:03.289997 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05ae507a-4ea9-4fa1-9d03-84dffe3af16f None admin] RBAC: Authorizing `identity:create_grant(project_id=e9919141e3b947ab8d91b55bba4a7de6, user_id=886eb5e00bbf4ec292eb962b665b37d0, role_id=9d9d5d2c270146488249b7ed943280db)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:03.291510 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05ae507a-4ea9-4fa1-9d03-84dffe3af16f None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:03.314029 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b348c646-7d76-4007-b4b2-2d9a3efc2d9a None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:03.315066 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 387/780] 10.222.0.77 () {58 vars in 1439 bytes} [Tue Aug 30 05:00:03 2022] PUT /identity/v3/projects/e9919141e3b947ab8d91b55bba4a7de6/users/886eb5e00bbf4ec292eb962b665b37d0/roles/9d9d5d2c270146488249b7ed943280db => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:00:03.316726 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b6bae57-f88e-4b06-b3e5-87e72dce3c41 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:03.317075 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b6bae57-f88e-4b06-b3e5-87e72dce3c41 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:03.317350 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b6bae57-f88e-4b06-b3e5-87e72dce3c41 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:03.350342 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-1b6bae57-f88e-4b06-b3e5-87e72dce3c41 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:00:03.351709 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-1b6bae57-f88e-4b06-b3e5-87e72dce3c41 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:00:03.476518 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc837645-ed51-4794-a6d3-7606fa12b347 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:03.477382 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc837645-ed51-4794-a6d3-7606fa12b347 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:03.477833 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc837645-ed51-4794-a6d3-7606fa12b347 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:03.478528 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc837645-ed51-4794-a6d3-7606fa12b347 None admin] PATH_INFO: `/v3/services` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:03.479198 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc837645-ed51-4794-a6d3-7606fa12b347 None admin] RBAC: Adding query filter params (type=network) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:03.479495 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc837645-ed51-4794-a6d3-7606fa12b347 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:03.480887 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc837645-ed51-4794-a6d3-7606fa12b347 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:03.490855 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 394/781] 10.222.0.77 () {56 vars in 1112 bytes} [Tue Aug 30 05:00:03 2022] GET /identity/v3/services?type=network => generated 350 bytes in 290 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:03.490855 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc837645-ed51-4794-a6d3-7606fa12b347 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:03.504172 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 388/782] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:00:03 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 05:00:03.505158 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b6bae57-f88e-4b06-b3e5-87e72dce3c41 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:03.561744 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:03.609542 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc837645-ed51-4794-a6d3-7606fa12b347 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:03.631155 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b6bae57-f88e-4b06-b3e5-87e72dce3c41 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:03.645597 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:03.650988 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8398a5d-ad3d-4302-8030-faa72fdded28 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:03.652207 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8398a5d-ad3d-4302-8030-faa72fdded28 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:03.652688 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8398a5d-ad3d-4302-8030-faa72fdded28 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:03.653158 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8398a5d-ad3d-4302-8030-faa72fdded28 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:03.666653 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8398a5d-ad3d-4302-8030-faa72fdded28 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 05:00:03.669125 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70e7c028-75bd-4463-bd1f-23a0fe578eed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:03.669732 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70e7c028-75bd-4463-bd1f-23a0fe578eed None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:03.670009 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70e7c028-75bd-4463-bd1f-23a0fe578eed None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:03.670333 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70e7c028-75bd-4463-bd1f-23a0fe578eed None admin] PATH_INFO: `/v3/endpoints` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:03.671070 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70e7c028-75bd-4463-bd1f-23a0fe578eed None admin] RBAC: Adding query filter params (interface=public, service_id=871cf3dfd2984c03bde94b40c5295f9e, region_id=RegionOne) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:03.671372 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70e7c028-75bd-4463-bd1f-23a0fe578eed None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:03.672729 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70e7c028-75bd-4463-bd1f-23a0fe578eed None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:03.674015 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-a8398a5d-ad3d-4302-8030-faa72fdded28 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 05:00:03.674015 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:03.674015 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:00:03.674015 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:00:03.674015 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:00:03.674015 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:00:03.674015 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:00:03.674015 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:00:03.674015 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:00:03.674015 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:00:03.674015 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:00:03.674015 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:00:03.674015 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 05:00:03.674015 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 05:00:03.674015 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 05:00:03.674015 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 05:00:03.675257 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 05:00:03.675257 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 05:00:03.675257 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 05:00:03.675257 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 05:00:03.675257 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:00:03.675257 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:00:03.675257 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:00:03.675257 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:00:03.675257 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:00:03.675257 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:00:03.675257 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:00:03.675257 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 05:00:03.675257 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:00:03.675257 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:00:03.675257 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:00:03.675257 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:00:03.676078 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:00:03.676078 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:00:03.676078 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:00:03.676078 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:00:03.676078 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 05:00:03.676078 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 05:00:03.676078 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 05:00:03.676078 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 05:00:03.676078 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 05:00:03.676078 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 05:00:03.676078 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 05:00:03.676078 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:00:03.682841 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 395/783] 10.222.0.77 () {56 vars in 1096 bytes} [Tue Aug 30 05:00:03 2022] GET /identity/v3/roles/admin => generated 83 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:00:03.685278 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8398a5d-ad3d-4302-8030-faa72fdded28 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:03.686129 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 389/784] 10.222.0.77 () {56 vars in 1251 bytes} [Tue Aug 30 05:00:03 2022] GET /identity/v3/endpoints?service_id=871cf3dfd2984c03bde94b40c5295f9e&interface=public®ion_id=RegionOne => generated 194 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:03.687399 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70e7c028-75bd-4463-bd1f-23a0fe578eed None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:03.797922 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8398a5d-ad3d-4302-8030-faa72fdded28 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:03.822258 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:03.835801 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89f1e030-3b43-4fa8-be4a-dbbfc87127be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:03.836848 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f1e030-3b43-4fa8-be4a-dbbfc87127be None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:03.837329 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f1e030-3b43-4fa8-be4a-dbbfc87127be None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:03.837795 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f1e030-3b43-4fa8-be4a-dbbfc87127be None admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:03.838649 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89f1e030-3b43-4fa8-be4a-dbbfc87127be None admin] RBAC: Adding query filter params (name=admin) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:03.839475 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89f1e030-3b43-4fa8-be4a-dbbfc87127be None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:03.841492 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89f1e030-3b43-4fa8-be4a-dbbfc87127be None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:03.852391 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70e7c028-75bd-4463-bd1f-23a0fe578eed None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:03.856458 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 396/785] 10.222.0.77 () {56 vars in 1099 bytes} [Tue Aug 30 05:00:03 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 05:00:03.861768 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89f1e030-3b43-4fa8-be4a-dbbfc87127be None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:03.896836 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-154ac946-7fa3-4ba6-bed2-9711123ee76a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:03.897919 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-154ac946-7fa3-4ba6-bed2-9711123ee76a None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:03.898727 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-154ac946-7fa3-4ba6-bed2-9711123ee76a None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:03.899623 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-154ac946-7fa3-4ba6-bed2-9711123ee76a None admin] PATH_INFO: `/v3/services` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:03.900697 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-154ac946-7fa3-4ba6-bed2-9711123ee76a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:03.902985 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-154ac946-7fa3-4ba6-bed2-9711123ee76a None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:03.911870 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 390/786] 10.222.0.77 () {56 vars in 1087 bytes} [Tue Aug 30 05:00:03 2022] GET /identity/v3/services => generated 772 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 05:00:03.912883 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-154ac946-7fa3-4ba6-bed2-9711123ee76a None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:03.998069 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89f1e030-3b43-4fa8-be4a-dbbfc87127be None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:04.020190 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:04.042930 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff0c6463-1164-45ee-b839-55b622b37986 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:04.043831 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff0c6463-1164-45ee-b839-55b622b37986 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:04.044273 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff0c6463-1164-45ee-b839-55b622b37986 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:04.044684 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff0c6463-1164-45ee-b839-55b622b37986 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:04.061232 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff0c6463-1164-45ee-b839-55b622b37986 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 05:00:04.066945 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-ff0c6463-1164-45ee-b839-55b622b37986 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 30 05:00:04.066945 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:04.066945 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:00:04.066945 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:00:04.066945 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:00:04.066945 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:00:04.066945 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:00:04.066945 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:00:04.066945 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:00:04.066945 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:00:04.066945 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:00:04.066945 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:00:04.066945 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 05:00:04.066945 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 05:00:04.066945 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 05:00:04.066945 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 05:00:04.068126 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 05:00:04.068126 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 05:00:04.068126 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 05:00:04.068126 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 05:00:04.068126 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:00:04.068126 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:00:04.068126 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:00:04.068126 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:00:04.068126 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:00:04.068126 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:00:04.068126 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:00:04.068126 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 05:00:04.068126 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:00:04.068126 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:00:04.068126 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:00:04.068126 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:00:04.069141 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:00:04.069141 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:00:04.069141 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:00:04.069141 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:00:04.069141 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 05:00:04.069141 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 05:00:04.069141 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 05:00:04.069141 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 05:00:04.069141 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 05:00:04.069141 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 05:00:04.069141 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 30 05:00:04.069141 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:00:04.071548 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 397/787] 10.222.0.77 () {56 vars in 1102 bytes} [Tue Aug 30 05:00:03 2022] GET /identity/v3/roles/service => generated 85 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:00:04.077340 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff0c6463-1164-45ee-b839-55b622b37986 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:04.115696 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-154ac946-7fa3-4ba6-bed2-9711123ee76a None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:04.152742 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:04.154330 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9503892b-1cb6-4528-873a-a4baa59d1fdb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:04.155012 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9503892b-1cb6-4528-873a-a4baa59d1fdb None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:04.155296 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9503892b-1cb6-4528-873a-a4baa59d1fdb None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:04.155593 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9503892b-1cb6-4528-873a-a4baa59d1fdb None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:04.168856 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9503892b-1cb6-4528-873a-a4baa59d1fdb None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:04.170464 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9503892b-1cb6-4528-873a-a4baa59d1fdb None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:04.187155 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-9503892b-1cb6-4528-873a-a4baa59d1fdb None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 05:00:04.187155 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:04.187155 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:00:04.187155 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:00:04.187155 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:00:04.187155 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:00:04.187155 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:00:04.187155 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:00:04.187155 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:00:04.187155 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:00:04.187155 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:00:04.187155 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:00:04.187155 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 05:00:04.187155 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 05:00:04.187155 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 05:00:04.187155 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 05:00:04.188148 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:00:04.188148 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:00:04.188148 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:00:04.188148 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:00:04.188148 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:00:04.188148 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:00:04.188148 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:00:04.188148 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 05:00:04.188148 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:00:04.188148 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:00:04.188148 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:00:04.188148 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:00:04.188148 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:00:04.188148 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:00:04.188148 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:00:04.188148 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:00:04.188148 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 05:00:04.189040 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 05:00:04.189040 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 05:00:04.189040 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:00:04.189040 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 391/788] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 05:00:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 276 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:00:04.195019 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9503892b-1cb6-4528-873a-a4baa59d1fdb None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:04.217938 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff0c6463-1164-45ee-b839-55b622b37986 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:04.254979 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f0f6cee-914a-4acf-be5f-9a945e396f8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:04.255693 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f0f6cee-914a-4acf-be5f-9a945e396f8a None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:04.255967 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f0f6cee-914a-4acf-be5f-9a945e396f8a None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:04.256236 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f0f6cee-914a-4acf-be5f-9a945e396f8a None admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:04.256892 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f0f6cee-914a-4acf-be5f-9a945e396f8a None admin] RBAC: Adding query filter params (name=service) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:04.257230 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f0f6cee-914a-4acf-be5f-9a945e396f8a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:04.258701 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f0f6cee-914a-4acf-be5f-9a945e396f8a None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:04.267714 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 398/789] 10.222.0.77 () {56 vars in 1103 bytes} [Tue Aug 30 05:00:04 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:04.273425 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f0f6cee-914a-4acf-be5f-9a945e396f8a None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:04.297869 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:04.307634 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9503892b-1cb6-4528-873a-a4baa59d1fdb None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:04.345827 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-657d21bf-a76f-4ab0-962a-ab7efa6276bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:04.346548 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-657d21bf-a76f-4ab0-962a-ab7efa6276bd None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:04.346924 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-657d21bf-a76f-4ab0-962a-ab7efa6276bd None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:04.347334 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-657d21bf-a76f-4ab0-962a-ab7efa6276bd None admin] PATH_INFO: `/v3/domains` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:04.348049 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-657d21bf-a76f-4ab0-962a-ab7efa6276bd None admin] RBAC: Adding query filter params (name=Default) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:04.348418 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-657d21bf-a76f-4ab0-962a-ab7efa6276bd None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:04.349859 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-657d21bf-a76f-4ab0-962a-ab7efa6276bd None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:04.364207 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 392/790] 10.222.0.77 () {56 vars in 1109 bytes} [Tue Aug 30 05:00:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 05:00:04.376044 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-657d21bf-a76f-4ab0-962a-ab7efa6276bd None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:04.377834 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f0f6cee-914a-4acf-be5f-9a945e396f8a None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:04.380176 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:04.414060 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfa5982e-6fd5-45b9-84c3-717680612a78 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:04.414678 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa5982e-6fd5-45b9-84c3-717680612a78 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:04.414978 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa5982e-6fd5-45b9-84c3-717680612a78 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:04.415247 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa5982e-6fd5-45b9-84c3-717680612a78 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:04.426528 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfa5982e-6fd5-45b9-84c3-717680612a78 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:04.427985 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfa5982e-6fd5-45b9-84c3-717680612a78 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:04.441934 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-cfa5982e-6fd5-45b9-84c3-717680612a78 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 05:00:04.441934 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:04.441934 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:00:04.441934 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:00:04.441934 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:00:04.441934 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:00:04.441934 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:00:04.441934 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:00:04.441934 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:00:04.441934 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:00:04.441934 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:00:04.441934 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:00:04.441934 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 05:00:04.441934 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 05:00:04.441934 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 05:00:04.441934 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 05:00:04.442913 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:00:04.442913 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:00:04.442913 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:00:04.442913 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:00:04.442913 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:00:04.442913 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:00:04.442913 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:00:04.442913 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 05:00:04.442913 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:00:04.442913 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:00:04.442913 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:00:04.442913 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:00:04.442913 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:00:04.442913 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:00:04.442913 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:00:04.442913 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:00:04.442913 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 05:00:04.443833 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 05:00:04.443833 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 05:00:04.443833 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:00:04.443833 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 399/791] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 05:00:04 2022] GET /identity/v3/domains/Default => generated 87 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 05:00:04.449501 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfa5982e-6fd5-45b9-84c3-717680612a78 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:04.570388 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-657d21bf-a76f-4ab0-962a-ab7efa6276bd None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:04.637001 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfa5982e-6fd5-45b9-84c3-717680612a78 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:04.643012 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66bc40f7-98cf-418d-8538-e28a52fda1c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:04.643644 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66bc40f7-98cf-418d-8538-e28a52fda1c0 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:04.643919 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66bc40f7-98cf-418d-8538-e28a52fda1c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:04.644186 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66bc40f7-98cf-418d-8538-e28a52fda1c0 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:04.656613 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66bc40f7-98cf-418d-8538-e28a52fda1c0 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:04.658082 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66bc40f7-98cf-418d-8538-e28a52fda1c0 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:04.683550 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-66bc40f7-98cf-418d-8538-e28a52fda1c0 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Aug 30 05:00:04.683550 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:04.683550 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:00:04.683550 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:00:04.683550 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:00:04.683550 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:00:04.683550 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:00:04.683550 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:00:04.683550 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:00:04.683550 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:00:04.683550 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:00:04.683550 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:00:04.683550 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 05:00:04.683550 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 05:00:04.683550 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 05:00:04.683550 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 05:00:04.684589 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:00:04.684589 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:00:04.684589 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 05:00:04.684589 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 05:00:04.684589 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 05:00:04.684589 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 05:00:04.684589 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:00:04.684589 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:00:04.684589 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:00:04.684589 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:00:04.684589 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:00:04.684589 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 05:00:04.684589 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:00:04.684589 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:00:04.684589 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:00:04.684589 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:00:04.684589 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:00:04.684589 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:00:04.685523 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:00:04.685523 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:00:04.685523 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 05:00:04.685523 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 05:00:04.685523 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 05:00:04.685523 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 05:00:04.685523 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 05:00:04.685523 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 05:00:04.685523 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Aug 30 05:00:04.685523 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:00:04.689091 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 393/792] 10.222.0.77 () {56 vars in 1096 bytes} [Tue Aug 30 05:00:04 2022] GET /identity/v3/users/swift => generated 83 bytes in 313 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:00:04.692346 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66bc40f7-98cf-418d-8538-e28a52fda1c0 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:04.714918 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f587811-ba06-48a2-8a9d-b1cd434155e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:04.715580 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f587811-ba06-48a2-8a9d-b1cd434155e1 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:04.715860 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f587811-ba06-48a2-8a9d-b1cd434155e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:04.716184 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f587811-ba06-48a2-8a9d-b1cd434155e1 None admin] PATH_INFO: `/v3/domains` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:04.716826 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f587811-ba06-48a2-8a9d-b1cd434155e1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:04.717120 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f587811-ba06-48a2-8a9d-b1cd434155e1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:04.718865 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f587811-ba06-48a2-8a9d-b1cd434155e1 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:04.719524 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:04.746710 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 400/793] 10.222.0.77 () {56 vars in 1109 bytes} [Tue Aug 30 05:00:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 297 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:04.749667 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f587811-ba06-48a2-8a9d-b1cd434155e1 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:04.779135 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:04.892513 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66bc40f7-98cf-418d-8538-e28a52fda1c0 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:04.935315 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f587811-ba06-48a2-8a9d-b1cd434155e1 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:04.958294 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd100e19-0d8a-439f-9e76-ac7207dcc51d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:04.958922 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd100e19-0d8a-439f-9e76-ac7207dcc51d None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:04.959207 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd100e19-0d8a-439f-9e76-ac7207dcc51d None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:04.959476 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd100e19-0d8a-439f-9e76-ac7207dcc51d None admin] PATH_INFO: `/v3/users` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:04.960253 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd100e19-0d8a-439f-9e76-ac7207dcc51d None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:04.960557 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd100e19-0d8a-439f-9e76-ac7207dcc51d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:04.961978 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd100e19-0d8a-439f-9e76-ac7207dcc51d None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:04.981325 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 394/794] 10.222.0.77 () {56 vars in 1135 bytes} [Tue Aug 30 05:00:04 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 289 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:04.986750 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd100e19-0d8a-439f-9e76-ac7207dcc51d None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:05.009834 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7090d4cd-5c2d-41f4-a5f8-c85b6c7c17cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:05.010501 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7090d4cd-5c2d-41f4-a5f8-c85b6c7c17cf None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:05.010778 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7090d4cd-5c2d-41f4-a5f8-c85b6c7c17cf None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:05.011066 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7090d4cd-5c2d-41f4-a5f8-c85b6c7c17cf None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:05.026550 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7090d4cd-5c2d-41f4-a5f8-c85b6c7c17cf None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:05.028994 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7090d4cd-5c2d-41f4-a5f8-c85b6c7c17cf None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:05.048343 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-7090d4cd-5c2d-41f4-a5f8-c85b6c7c17cf None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 30 05:00:05.048343 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:05.048343 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:00:05.048343 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:00:05.048343 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:00:05.048343 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:00:05.048343 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:00:05.048343 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:00:05.048343 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:00:05.048343 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:00:05.048343 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:00:05.048343 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:00:05.048343 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 05:00:05.048343 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 05:00:05.048343 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 05:00:05.048343 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 05:00:05.049309 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:00:05.049309 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:00:05.049309 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 05:00:05.049309 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 05:00:05.049309 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 05:00:05.049309 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 05:00:05.049309 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:00:05.049309 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:00:05.049309 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:00:05.049309 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:00:05.049309 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:00:05.049309 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 05:00:05.049309 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:00:05.049309 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:00:05.049309 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:00:05.049309 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:00:05.049309 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:00:05.049309 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:00:05.050238 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:00:05.050238 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:00:05.050238 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 05:00:05.050238 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 05:00:05.050238 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 05:00:05.050238 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 05:00:05.050238 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 05:00:05.050238 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 05:00:05.050238 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 30 05:00:05.050238 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:00:05.054224 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 401/795] 10.222.0.77 () {56 vars in 1117 bytes} [Tue Aug 30 05:00:04 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 305 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:00:05.059771 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7090d4cd-5c2d-41f4-a5f8-c85b6c7c17cf None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:05.092256 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:05.148111 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd100e19-0d8a-439f-9e76-ac7207dcc51d None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:05.177051 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:05.187454 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41a91705-04d6-4eeb-b62d-f5786d2a3ac8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:05.188375 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41a91705-04d6-4eeb-b62d-f5786d2a3ac8 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:05.188798 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41a91705-04d6-4eeb-b62d-f5786d2a3ac8 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:05.189215 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41a91705-04d6-4eeb-b62d-f5786d2a3ac8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:05.201470 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41a91705-04d6-4eeb-b62d-f5786d2a3ac8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:05.203027 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41a91705-04d6-4eeb-b62d-f5786d2a3ac8 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:05.208055 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7090d4cd-5c2d-41f4-a5f8-c85b6c7c17cf None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:05.218532 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-41a91705-04d6-4eeb-b62d-f5786d2a3ac8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 05:00:05.218532 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:05.218532 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:00:05.218532 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:00:05.218532 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:00:05.218532 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:00:05.218532 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:00:05.218532 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:00:05.218532 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:00:05.218532 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:00:05.218532 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:00:05.218532 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:00:05.218532 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 05:00:05.218532 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 05:00:05.218532 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 05:00:05.218532 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 05:00:05.219604 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:00:05.219604 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:00:05.219604 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:00:05.219604 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:00:05.219604 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:00:05.219604 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:00:05.219604 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:00:05.219604 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 05:00:05.219604 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:00:05.219604 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:00:05.219604 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:00:05.219604 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:00:05.219604 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:00:05.219604 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:00:05.219604 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:00:05.219604 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:00:05.219604 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 05:00:05.220472 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 05:00:05.220472 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 05:00:05.220472 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:00:05.220472 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 395/796] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 05:00:04 2022] GET /identity/v3/domains/Default => generated 87 bytes in 234 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:00:05.226807 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23f23042-5e4e-446d-b156-d7a32f38df47 None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:05.226807 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23f23042-5e4e-446d-b156-d7a32f38df47 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:05.226807 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23f23042-5e4e-446d-b156-d7a32f38df47 None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:05.226807 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 396/797] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:00:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:00:05.226807 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23f23042-5e4e-446d-b156-d7a32f38df47 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:05.259675 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-173ba33d-3f0d-4000-9f30-52b60ee7d5c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:05.260654 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-173ba33d-3f0d-4000-9f30-52b60ee7d5c1 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:05.261107 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-173ba33d-3f0d-4000-9f30-52b60ee7d5c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:05.261583 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-173ba33d-3f0d-4000-9f30-52b60ee7d5c1 None admin] PATH_INFO: `/v3/users` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:05.262467 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-173ba33d-3f0d-4000-9f30-52b60ee7d5c1 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:05.262941 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-173ba33d-3f0d-4000-9f30-52b60ee7d5c1 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:05.264726 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-173ba33d-3f0d-4000-9f30-52b60ee7d5c1 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:05.286679 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 402/798] 10.222.0.77 () {56 vars in 1149 bytes} [Tue Aug 30 05:00:05 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:05.288204 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84337007-04aa-45c0-ade9-b288011bd222 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:05.288646 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84337007-04aa-45c0-ade9-b288011bd222 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:05.289054 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84337007-04aa-45c0-ade9-b288011bd222 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:05.315122 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-84337007-04aa-45c0-ade9-b288011bd222 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:00:05.316739 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-84337007-04aa-45c0-ade9-b288011bd222 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:00:05.359265 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23f23042-5e4e-446d-b156-d7a32f38df47 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:05.388644 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:05.407214 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 403/799] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:00:05 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 05:00:05.407214 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84337007-04aa-45c0-ade9-b288011bd222 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:05.410201 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7a6707e-bc1f-4e76-a8e2-31132418035a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:05.410801 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7a6707e-bc1f-4e76-a8e2-31132418035a None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:05.411098 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7a6707e-bc1f-4e76-a8e2-31132418035a None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:05.411366 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7a6707e-bc1f-4e76-a8e2-31132418035a None admin] PATH_INFO: `/v3/domains` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:05.412017 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7a6707e-bc1f-4e76-a8e2-31132418035a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:05.412313 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7a6707e-bc1f-4e76-a8e2-31132418035a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:05.413725 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7a6707e-bc1f-4e76-a8e2-31132418035a None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:05.446125 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 397/800] 10.222.0.77 () {56 vars in 1109 bytes} [Tue Aug 30 05:00:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:05.450281 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7a6707e-bc1f-4e76-a8e2-31132418035a None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:05.507426 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:05.551774 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84337007-04aa-45c0-ade9-b288011bd222 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:05.591751 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2deb766e-f473-473b-8e36-69dee78becd3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:05.593289 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2deb766e-f473-473b-8e36-69dee78becd3 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:05.593859 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2deb766e-f473-473b-8e36-69dee78becd3 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:05.594334 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2deb766e-f473-473b-8e36-69dee78becd3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:05.607415 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2deb766e-f473-473b-8e36-69dee78becd3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:05.609551 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2deb766e-f473-473b-8e36-69dee78becd3 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:05.629130 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-2deb766e-f473-473b-8e36-69dee78becd3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 05:00:05.629130 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:05.629130 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:00:05.629130 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:00:05.629130 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:00:05.629130 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:00:05.629130 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:00:05.629130 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:00:05.629130 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:00:05.629130 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:00:05.629130 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:00:05.629130 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:00:05.629130 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 05:00:05.629130 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 05:00:05.629130 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 05:00:05.629130 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 05:00:05.630056 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:00:05.630056 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:00:05.630056 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:00:05.630056 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:00:05.630056 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:00:05.630056 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:00:05.630056 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:00:05.630056 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 05:00:05.630056 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:00:05.630056 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:00:05.630056 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:00:05.630056 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:00:05.630056 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:00:05.630056 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:00:05.630056 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:00:05.630056 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:00:05.630056 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 05:00:05.630973 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 05:00:05.630973 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 05:00:05.630973 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:00:05.638106 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 404/801] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 05:00:05 2022] GET /identity/v3/domains/Default => generated 87 bytes in 230 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:00:05.639628 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e13f846-76b1-4a3a-97fd-6cdab3edc8ca None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:05.640111 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e13f846-76b1-4a3a-97fd-6cdab3edc8ca None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:05.640604 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e13f846-76b1-4a3a-97fd-6cdab3edc8ca None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:05.642703 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 405/802] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:00:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:00:05.643709 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e13f846-76b1-4a3a-97fd-6cdab3edc8ca None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:05.653195 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7a6707e-bc1f-4e76-a8e2-31132418035a None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:05.688936 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8f25ee0-45a6-4a77-b565-c764efcc7c4a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:05.690060 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8f25ee0-45a6-4a77-b565-c764efcc7c4a None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:05.690060 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8f25ee0-45a6-4a77-b565-c764efcc7c4a None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:05.690060 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8f25ee0-45a6-4a77-b565-c764efcc7c4a None admin] PATH_INFO: `/v3/projects/service` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:05.697459 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8f25ee0-45a6-4a77-b565-c764efcc7c4a None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:05.698987 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8f25ee0-45a6-4a77-b565-c764efcc7c4a None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:05.711478 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-b8f25ee0-45a6-4a77-b565-c764efcc7c4a None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 05:00:05.711478 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:05.711478 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:00:05.711478 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:00:05.711478 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:00:05.711478 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:00:05.711478 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:00:05.711478 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:00:05.711478 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:00:05.711478 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:00:05.711478 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:00:05.711478 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:00:05.711478 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 05:00:05.711478 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 05:00:05.711478 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 05:00:05.712366 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 05:00:05.712366 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:00:05.712366 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:00:05.712366 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:00:05.712366 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:00:05.712366 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:00:05.712366 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:00:05.712366 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:00:05.712366 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 05:00:05.712366 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:00:05.712366 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:00:05.712366 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:00:05.712366 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:00:05.712366 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:00:05.712366 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:00:05.712366 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:00:05.712366 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:00:05.713812 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 05:00:05.713812 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 05:00:05.713812 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 05:00:05.713812 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 05:00:05.713812 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 05:00:05.713812 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 05:00:05.713812 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 05:00:05.713812 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:00:05.713812 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 398/803] 10.222.0.77 () {56 vars in 1111 bytes} [Tue Aug 30 05:00:05 2022] GET /identity/v3/projects/service => generated 88 bytes in 263 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 05:00:05.714292 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e42926-d9bb-4e7a-837b-55654faa982c None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:05.714358 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e42926-d9bb-4e7a-837b-55654faa982c None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:05.714598 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e42926-d9bb-4e7a-837b-55654faa982c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:05.732959 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-b2e42926-d9bb-4e7a-837b-55654faa982c None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:00:05.734127 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-b2e42926-d9bb-4e7a-837b-55654faa982c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:00:05.770599 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e13f846-76b1-4a3a-97fd-6cdab3edc8ca None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:05.773247 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:05.785646 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:05.806433 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14048fa1-a069-41cd-b250-7dce134480b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:05.807138 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14048fa1-a069-41cd-b250-7dce134480b3 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:05.807417 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14048fa1-a069-41cd-b250-7dce134480b3 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:05.807684 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14048fa1-a069-41cd-b250-7dce134480b3 None admin] PATH_INFO: `/v3/domains` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:05.808321 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14048fa1-a069-41cd-b250-7dce134480b3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:05.819172 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 399/804] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:00:05 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 05:00:05.819883 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14048fa1-a069-41cd-b250-7dce134480b3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:05.819993 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2e42926-d9bb-4e7a-837b-55654faa982c None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:05.820069 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14048fa1-a069-41cd-b250-7dce134480b3 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:05.831123 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 406/805] 10.222.0.77 () {56 vars in 1109 bytes} [Tue Aug 30 05:00:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:05.831959 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14048fa1-a069-41cd-b250-7dce134480b3 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:05.931470 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2e42926-d9bb-4e7a-837b-55654faa982c None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:05.952572 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14048fa1-a069-41cd-b250-7dce134480b3 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:05.982009 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c731404e-59b3-4ea8-a659-7f4e35cafb7f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:05.982900 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c731404e-59b3-4ea8-a659-7f4e35cafb7f None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:05.983181 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c731404e-59b3-4ea8-a659-7f4e35cafb7f None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:05.983452 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c731404e-59b3-4ea8-a659-7f4e35cafb7f None admin] PATH_INFO: `/v3/projects` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:05.984771 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c731404e-59b3-4ea8-a659-7f4e35cafb7f None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:05.985079 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c731404e-59b3-4ea8-a659-7f4e35cafb7f None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:05.986845 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c731404e-59b3-4ea8-a659-7f4e35cafb7f None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:05.989444 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44381b7c-e0b7-4bc7-ae83-f2445dee2c47 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:05.990120 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44381b7c-e0b7-4bc7-ae83-f2445dee2c47 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:05.990400 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44381b7c-e0b7-4bc7-ae83-f2445dee2c47 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:05.990669 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44381b7c-e0b7-4bc7-ae83-f2445dee2c47 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:05.991827 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44381b7c-e0b7-4bc7-ae83-f2445dee2c47 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:05.993168 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44381b7c-e0b7-4bc7-ae83-f2445dee2c47 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:06.000227 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 400/806] 10.222.0.77 () {56 vars in 1148 bytes} [Tue Aug 30 05:00:05 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:06.001230 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c731404e-59b3-4ea8-a659-7f4e35cafb7f None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:06.007767 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-44381b7c-e0b7-4bc7-ae83-f2445dee2c47 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 30 05:00:06.007767 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:06.007767 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 30 05:00:06.007767 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 30 05:00:06.007767 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 30 05:00:06.007767 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 30 05:00:06.007767 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 30 05:00:06.007767 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 05:00:06.007767 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 30 05:00:06.007767 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:00:06.007767 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 05:00:06.007767 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:00:06.007767 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:06.007767 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:00:06.007767 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:00:06.007767 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:00:06.007767 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:00:06.008741 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:00:06.008741 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:00:06.008741 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:00:06.008741 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:00:06.008741 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:00:06.008741 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:00:06.008741 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 30 05:00:06.008741 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 30 05:00:06.008741 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 30 05:00:06.008741 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 30 05:00:06.008741 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:00:06.008741 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:00:06.008741 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:00:06.008741 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:00:06.008741 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:00:06.008741 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:00:06.009558 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:00:06.009558 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 05:00:06.009558 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:00:06.009558 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:00:06.009558 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:00:06.009558 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:00:06.009558 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:00:06.009558 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:00:06.009558 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:00:06.009558 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:00:06.009558 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 30 05:00:06.009558 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 05:00:06.009558 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 30 05:00:06.009558 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:00:06.013849 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 407/807] 10.222.0.77 () {56 vars in 1129 bytes} [Tue Aug 30 05:00:05 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:00:06.014733 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44381b7c-e0b7-4bc7-ae83-f2445dee2c47 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:06.036816 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:06.061296 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:06.130648 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c731404e-59b3-4ea8-a659-7f4e35cafb7f None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:06.135267 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44381b7c-e0b7-4bc7-ae83-f2445dee2c47 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:06.171574 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e15437-2de0-409b-a2ed-bda06a9ad1ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:06.171782 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe9d684b-1ef8-41b2-b8bd-2b911aeb7021 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:06.172248 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe9d684b-1ef8-41b2-b8bd-2b911aeb7021 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:06.172532 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe9d684b-1ef8-41b2-b8bd-2b911aeb7021 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:06.172799 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe9d684b-1ef8-41b2-b8bd-2b911aeb7021 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:06.173354 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e15437-2de0-409b-a2ed-bda06a9ad1ea None admin] REQUEST_METHOD: `PUT` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:06.173771 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e15437-2de0-409b-a2ed-bda06a9ad1ea None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:06.174189 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e15437-2de0-409b-a2ed-bda06a9ad1ea None admin] PATH_INFO: `/v3/projects/e9919141e3b947ab8d91b55bba4a7de6/users/e0a7f7afc3d44ec788c05d005e52a1b8/roles/9d9d5d2c270146488249b7ed943280db` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:06.176552 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0e15437-2de0-409b-a2ed-bda06a9ad1ea None admin] RBAC: Authorizing `identity:create_grant(project_id=e9919141e3b947ab8d91b55bba4a7de6, user_id=e0a7f7afc3d44ec788c05d005e52a1b8, role_id=9d9d5d2c270146488249b7ed943280db)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:06.178352 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0e15437-2de0-409b-a2ed-bda06a9ad1ea None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:06.180169 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe9d684b-1ef8-41b2-b8bd-2b911aeb7021 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:06.181774 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe9d684b-1ef8-41b2-b8bd-2b911aeb7021 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:06.200799 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-fe9d684b-1ef8-41b2-b8bd-2b911aeb7021 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 05:00:06.200799 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:06.200799 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:00:06.200799 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:00:06.200799 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:00:06.200799 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:00:06.200799 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:00:06.200799 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:00:06.200799 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:00:06.200799 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:00:06.200799 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:00:06.200799 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:00:06.200799 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 05:00:06.200799 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 05:00:06.200799 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 05:00:06.201982 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 05:00:06.201982 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:00:06.201982 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:00:06.201982 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:00:06.201982 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:00:06.201982 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:00:06.201982 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:00:06.201982 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:00:06.201982 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 05:00:06.201982 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:00:06.201982 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:00:06.201982 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:00:06.201982 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:00:06.201982 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:00:06.201982 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:00:06.201982 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:00:06.201982 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:00:06.203151 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 05:00:06.203151 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 05:00:06.203151 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 05:00:06.203151 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 05:00:06.203151 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 05:00:06.203151 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 05:00:06.203151 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 05:00:06.203151 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:00:06.203151 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 401/809] 10.222.0.77 () {56 vars in 1111 bytes} [Tue Aug 30 05:00:06 2022] GET /identity/v3/projects/service => generated 88 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:00:06.203151 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe9d684b-1ef8-41b2-b8bd-2b911aeb7021 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:06.204118 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 408/809] 10.222.0.77 () {58 vars in 1439 bytes} [Tue Aug 30 05:00:06 2022] PUT /identity/v3/projects/e9919141e3b947ab8d91b55bba4a7de6/users/e0a7f7afc3d44ec788c05d005e52a1b8/roles/9d9d5d2c270146488249b7ed943280db => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:00:06.205358 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e15437-2de0-409b-a2ed-bda06a9ad1ea None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:06.348367 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:06.357330 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e15437-2de0-409b-a2ed-bda06a9ad1ea None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:06.369418 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:06.386917 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe9d684b-1ef8-41b2-b8bd-2b911aeb7021 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:06.403878 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b136332-a576-4ede-9b03-2c8b91b07ca7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:06.404673 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b136332-a576-4ede-9b03-2c8b91b07ca7 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:06.405027 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b136332-a576-4ede-9b03-2c8b91b07ca7 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:06.405539 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b136332-a576-4ede-9b03-2c8b91b07ca7 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:06.416666 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b136332-a576-4ede-9b03-2c8b91b07ca7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 05:00:06.421006 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae5db5e6-809b-4eba-91b5-ae9c87dfff2f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:06.421638 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae5db5e6-809b-4eba-91b5-ae9c87dfff2f None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:06.421918 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae5db5e6-809b-4eba-91b5-ae9c87dfff2f None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:06.422187 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae5db5e6-809b-4eba-91b5-ae9c87dfff2f None admin] PATH_INFO: `/v3/services` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:06.422449 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-8b136332-a576-4ede-9b03-2c8b91b07ca7 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 05:00:06.422449 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:06.422449 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:00:06.422449 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:00:06.422449 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:00:06.422449 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:00:06.422449 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:00:06.422449 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:00:06.422449 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:00:06.422449 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:00:06.422449 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:00:06.422449 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:00:06.422449 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 05:00:06.422449 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 05:00:06.422449 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 05:00:06.422449 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 05:00:06.423437 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 05:00:06.423437 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 05:00:06.423437 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 05:00:06.423437 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 05:00:06.423437 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:00:06.423437 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:00:06.423437 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:00:06.423437 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:00:06.423437 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:00:06.423437 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:00:06.423437 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:00:06.423437 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 05:00:06.423437 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:00:06.423437 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:00:06.423437 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:00:06.423437 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:00:06.424268 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:00:06.424268 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:00:06.424268 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:00:06.424268 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:00:06.424268 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 05:00:06.424268 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 05:00:06.424268 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 05:00:06.424268 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 05:00:06.424268 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 05:00:06.424268 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 05:00:06.424268 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 05:00:06.424268 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:00:06.424893 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae5db5e6-809b-4eba-91b5-ae9c87dfff2f None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:06.424893 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae5db5e6-809b-4eba-91b5-ae9c87dfff2f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:06.425013 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 409/810] 10.222.0.77 () {56 vars in 1096 bytes} [Tue Aug 30 05:00:06 2022] GET /identity/v3/roles/admin => generated 83 bytes in 219 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:00:06.425589 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae5db5e6-809b-4eba-91b5-ae9c87dfff2f None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:06.425758 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b136332-a576-4ede-9b03-2c8b91b07ca7 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:06.429300 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 402/811] 10.222.0.77 () {56 vars in 1124 bytes} [Tue Aug 30 05:00:06 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:06.435087 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb633089-3838-4883-ae54-fa31a9fa9a77 None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:06.435377 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb633089-3838-4883-ae54-fa31a9fa9a77 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:06.435652 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb633089-3838-4883-ae54-fa31a9fa9a77 None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:06.437039 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 403/812] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:00:06 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:00:06.437591 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb633089-3838-4883-ae54-fa31a9fa9a77 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:06.579127 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b136332-a576-4ede-9b03-2c8b91b07ca7 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:06.593658 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb633089-3838-4883-ae54-fa31a9fa9a77 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:06.615371 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f2aaa4e-5a71-49bd-96c0-4865a8e4b751 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:06.616421 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f2aaa4e-5a71-49bd-96c0-4865a8e4b751 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:06.616699 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f2aaa4e-5a71-49bd-96c0-4865a8e4b751 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:06.617126 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f2aaa4e-5a71-49bd-96c0-4865a8e4b751 None admin] PATH_INFO: `/v3/projects` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:06.617819 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f2aaa4e-5a71-49bd-96c0-4865a8e4b751 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:06.618121 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f2aaa4e-5a71-49bd-96c0-4865a8e4b751 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:06.619573 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f2aaa4e-5a71-49bd-96c0-4865a8e4b751 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:06.629904 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3573177-a356-4384-8a0b-825ed9b03fe0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:06.630507 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3573177-a356-4384-8a0b-825ed9b03fe0 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:06.630857 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3573177-a356-4384-8a0b-825ed9b03fe0 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:06.631139 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3573177-a356-4384-8a0b-825ed9b03fe0 None admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:06.631852 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3573177-a356-4384-8a0b-825ed9b03fe0 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:06.637124 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3573177-a356-4384-8a0b-825ed9b03fe0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:06.637258 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 410/813] 10.222.0.77 () {56 vars in 1148 bytes} [Tue Aug 30 05:00:06 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:06.637258 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f2aaa4e-5a71-49bd-96c0-4865a8e4b751 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:06.637438 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3573177-a356-4384-8a0b-825ed9b03fe0 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:06.648439 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 404/814] 10.222.0.77 () {56 vars in 1099 bytes} [Tue Aug 30 05:00:06 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:06.650276 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-378d96c8-2c18-4589-ba40-f9c108f20438 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:06.651047 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-378d96c8-2c18-4589-ba40-f9c108f20438 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:06.651760 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-378d96c8-2c18-4589-ba40-f9c108f20438 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:06.658067 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:06.673852 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-378d96c8-2c18-4589-ba40-f9c108f20438 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:00:06.675099 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-378d96c8-2c18-4589-ba40-f9c108f20438 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:00:06.690436 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:06.743440 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f2aaa4e-5a71-49bd-96c0-4865a8e4b751 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:06.758848 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 405/815] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:00:06 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 05:00:06.758848 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-378d96c8-2c18-4589-ba40-f9c108f20438 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:06.796034 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea04b555-03d6-49b8-a76b-2ae21e9ae4e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:06.796684 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea04b555-03d6-49b8-a76b-2ae21e9ae4e8 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:06.796908 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea04b555-03d6-49b8-a76b-2ae21e9ae4e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:06.797178 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea04b555-03d6-49b8-a76b-2ae21e9ae4e8 None admin] PATH_INFO: `/v3/services` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:06.797801 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea04b555-03d6-49b8-a76b-2ae21e9ae4e8 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:06.798100 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea04b555-03d6-49b8-a76b-2ae21e9ae4e8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:06.799721 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea04b555-03d6-49b8-a76b-2ae21e9ae4e8 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:06.809112 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 411/816] 10.222.0.77 () {56 vars in 1124 bytes} [Tue Aug 30 05:00:06 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:06.809112 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea04b555-03d6-49b8-a76b-2ae21e9ae4e8 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:06.896797 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-378d96c8-2c18-4589-ba40-f9c108f20438 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:06.920750 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:06.940633 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2bcb401-85a8-42ee-ba4b-71f52a853873 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:06.941415 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2bcb401-85a8-42ee-ba4b-71f52a853873 None admin] REQUEST_METHOD: `PUT` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:06.941956 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2bcb401-85a8-42ee-ba4b-71f52a853873 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:06.942333 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2bcb401-85a8-42ee-ba4b-71f52a853873 None admin] PATH_INFO: `/v3/projects/e9919141e3b947ab8d91b55bba4a7de6/users/fe51dbec91624ed1ab00b441e29a3dd5/roles/821647e8b40542499667e29e1b301cb2` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:06.946662 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea04b555-03d6-49b8-a76b-2ae21e9ae4e8 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:06.961982 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2bcb401-85a8-42ee-ba4b-71f52a853873 None admin] RBAC: Authorizing `identity:create_grant(project_id=e9919141e3b947ab8d91b55bba4a7de6, user_id=fe51dbec91624ed1ab00b441e29a3dd5, role_id=821647e8b40542499667e29e1b301cb2)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:06.963444 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2bcb401-85a8-42ee-ba4b-71f52a853873 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:06.972198 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:06.986847 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 406/817] 10.222.0.77 () {58 vars in 1439 bytes} [Tue Aug 30 05:00:06 2022] PUT /identity/v3/projects/e9919141e3b947ab8d91b55bba4a7de6/users/fe51dbec91624ed1ab00b441e29a3dd5/roles/821647e8b40542499667e29e1b301cb2 => generated 0 bytes in 228 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:00:06.986847 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2bcb401-85a8-42ee-ba4b-71f52a853873 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:07.008389 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ca98c12-1957-46f4-a128-c4498fa5e1a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:07.009446 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca98c12-1957-46f4-a128-c4498fa5e1a0 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:07.010115 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca98c12-1957-46f4-a128-c4498fa5e1a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:07.010601 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca98c12-1957-46f4-a128-c4498fa5e1a0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:07.043736 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ca98c12-1957-46f4-a128-c4498fa5e1a0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:07.045239 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ca98c12-1957-46f4-a128-c4498fa5e1a0 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:07.062519 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-0ca98c12-1957-46f4-a128-c4498fa5e1a0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 05:00:07.062519 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:07.062519 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:00:07.062519 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:00:07.062519 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:00:07.062519 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:00:07.062519 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:00:07.062519 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:00:07.062519 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:00:07.062519 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:00:07.062519 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:00:07.062519 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:00:07.062519 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 05:00:07.062519 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 05:00:07.062519 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 05:00:07.062519 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 05:00:07.063620 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:00:07.063620 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:00:07.063620 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:00:07.063620 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:00:07.063620 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:00:07.063620 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:00:07.063620 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:00:07.063620 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 05:00:07.063620 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:00:07.063620 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:00:07.063620 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:00:07.063620 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:00:07.063620 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:00:07.063620 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:00:07.063620 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:00:07.063620 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:00:07.063620 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 05:00:07.064515 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 05:00:07.064515 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 05:00:07.064515 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:00:07.068945 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 412/818] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 05:00:06 2022] GET /identity/v3/domains/Default => generated 87 bytes in 258 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:00:07.071341 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ca98c12-1957-46f4-a128-c4498fa5e1a0 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:07.105129 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2bcb401-85a8-42ee-ba4b-71f52a853873 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:07.145940 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf4f01d5-988b-439b-8b3a-a3d6029beaa1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:07.146894 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf4f01d5-988b-439b-8b3a-a3d6029beaa1 None admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:07.147405 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf4f01d5-988b-439b-8b3a-a3d6029beaa1 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:07.148301 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf4f01d5-988b-439b-8b3a-a3d6029beaa1 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:07.149170 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf4f01d5-988b-439b-8b3a-a3d6029beaa1 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:07.150779 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf4f01d5-988b-439b-8b3a-a3d6029beaa1 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:07.179966 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 407/819] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:00:06 2022] POST /identity/v3/endpoints => generated 346 bytes in 200 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:00:07.180621 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf4f01d5-988b-439b-8b3a-a3d6029beaa1 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:07.196293 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ca98c12-1957-46f4-a128-c4498fa5e1a0 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:07.240458 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:07.246145 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ed181f9-e98c-477c-93ce-edb01d756bbd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:07.246803 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed181f9-e98c-477c-93ce-edb01d756bbd None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:07.247134 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed181f9-e98c-477c-93ce-edb01d756bbd None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:07.247376 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed181f9-e98c-477c-93ce-edb01d756bbd None admin] PATH_INFO: `/v3/services/network` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:07.248006 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ed181f9-e98c-477c-93ce-edb01d756bbd None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:07.249530 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ed181f9-e98c-477c-93ce-edb01d756bbd None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:07.261959 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:07.262166 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-5ed181f9-e98c-477c-93ce-edb01d756bbd None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Aug 30 05:00:07.262166 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:07.262166 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 30 05:00:07.262166 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 30 05:00:07.262166 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 30 05:00:07.262166 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 30 05:00:07.262166 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 30 05:00:07.262166 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 05:00:07.262166 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Aug 30 05:00:07.262166 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:00:07.262166 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 05:00:07.262166 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:00:07.262166 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:07.262166 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:00:07.262166 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:00:07.262166 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:00:07.262166 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:00:07.263237 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:00:07.263237 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:00:07.263237 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:00:07.263237 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:00:07.263237 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:00:07.263237 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:00:07.263237 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 30 05:00:07.263237 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 30 05:00:07.263237 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 30 05:00:07.263237 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 30 05:00:07.263237 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:00:07.263237 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:00:07.263237 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:00:07.263237 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:00:07.263237 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:00:07.263237 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:00:07.264401 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:00:07.264401 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 05:00:07.264401 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:00:07.264401 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:00:07.264401 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:00:07.264401 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:00:07.264401 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:00:07.264401 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:00:07.264401 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:00:07.264401 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:00:07.264401 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 30 05:00:07.264401 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 05:00:07.264401 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Aug 30 05:00:07.264401 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:00:07.264401 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 413/820] 10.222.0.77 () {56 vars in 1111 bytes} [Tue Aug 30 05:00:07 2022] GET /identity/v3/services/network => generated 88 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:00:07.269447 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ed181f9-e98c-477c-93ce-edb01d756bbd None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:07.338348 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf4f01d5-988b-439b-8b3a-a3d6029beaa1 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:07.410079 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-161a6f0e-cfcf-48b6-81f2-0b82f04b0c35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:07.410757 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-161a6f0e-cfcf-48b6-81f2-0b82f04b0c35 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:07.411072 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-161a6f0e-cfcf-48b6-81f2-0b82f04b0c35 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:07.411529 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-161a6f0e-cfcf-48b6-81f2-0b82f04b0c35 None admin] PATH_INFO: `/v3/domains` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:07.412221 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-161a6f0e-cfcf-48b6-81f2-0b82f04b0c35 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:07.412533 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-161a6f0e-cfcf-48b6-81f2-0b82f04b0c35 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:07.414108 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-161a6f0e-cfcf-48b6-81f2-0b82f04b0c35 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:07.427524 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 408/821] 10.222.0.77 () {56 vars in 1109 bytes} [Tue Aug 30 05:00:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 247 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:07.433264 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ed181f9-e98c-477c-93ce-edb01d756bbd None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:07.438550 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-161a6f0e-cfcf-48b6-81f2-0b82f04b0c35 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:07.471468 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd1e2e6-452a-408e-9655-a7972f9cc9b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:07.472075 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd1e2e6-452a-408e-9655-a7972f9cc9b1 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:07.472505 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd1e2e6-452a-408e-9655-a7972f9cc9b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:07.472711 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd1e2e6-452a-408e-9655-a7972f9cc9b1 None admin] PATH_INFO: `/v3/services` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:07.473408 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffd1e2e6-452a-408e-9655-a7972f9cc9b1 None admin] RBAC: Adding query filter params (name=network) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:07.473710 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffd1e2e6-452a-408e-9655-a7972f9cc9b1 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:07.475836 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffd1e2e6-452a-408e-9655-a7972f9cc9b1 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:07.481523 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 414/822] 10.222.0.77 () {56 vars in 1112 bytes} [Tue Aug 30 05:00:07 2022] GET /identity/v3/services?name=network => generated 124 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:07.486853 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd1e2e6-452a-408e-9655-a7972f9cc9b1 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:07.549481 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-161a6f0e-cfcf-48b6-81f2-0b82f04b0c35 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:07.550862 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:07.565476 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:07.591964 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9dcc2f5-6c11-4490-a629-155bc5bfeb79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:07.592612 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9dcc2f5-6c11-4490-a629-155bc5bfeb79 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:07.592889 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9dcc2f5-6c11-4490-a629-155bc5bfeb79 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:07.593157 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9dcc2f5-6c11-4490-a629-155bc5bfeb79 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:07.599573 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd1e2e6-452a-408e-9655-a7972f9cc9b1 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:07.600897 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9dcc2f5-6c11-4490-a629-155bc5bfeb79 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:07.602352 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9dcc2f5-6c11-4490-a629-155bc5bfeb79 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:07.615832 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-d9dcc2f5-6c11-4490-a629-155bc5bfeb79 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 30 05:00:07.615832 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:07.615832 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:00:07.615832 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:00:07.615832 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:00:07.615832 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:00:07.615832 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:00:07.615832 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:00:07.615832 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:00:07.615832 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:00:07.615832 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:00:07.615832 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:00:07.615832 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 05:00:07.615832 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 05:00:07.615832 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 05:00:07.615832 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 05:00:07.617348 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:00:07.617348 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:00:07.617348 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 05:00:07.617348 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 05:00:07.617348 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 05:00:07.617348 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 05:00:07.617348 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:00:07.617348 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:00:07.617348 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:00:07.617348 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:00:07.617348 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:00:07.617348 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 05:00:07.617348 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:00:07.617348 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:00:07.617348 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:00:07.617348 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:00:07.617348 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:00:07.617348 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:00:07.618287 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:00:07.618287 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:00:07.618287 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 05:00:07.618287 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 05:00:07.618287 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 05:00:07.618287 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 05:00:07.618287 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 05:00:07.618287 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 05:00:07.618287 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 30 05:00:07.618287 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:00:07.618287 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 409/823] 10.222.0.77 () {56 vars in 1093 bytes} [Tue Aug 30 05:00:07 2022] GET /identity/v3/users/nova => generated 82 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:00:07.623615 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9dcc2f5-6c11-4490-a629-155bc5bfeb79 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:07.637681 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d31a379-2816-4f02-86f7-b4d2925d09f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:07.638292 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d31a379-2816-4f02-86f7-b4d2925d09f6 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:07.638568 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d31a379-2816-4f02-86f7-b4d2925d09f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:07.638862 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d31a379-2816-4f02-86f7-b4d2925d09f6 None admin] PATH_INFO: `/v3/services` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:07.639495 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d31a379-2816-4f02-86f7-b4d2925d09f6 None admin] RBAC: Adding query filter params (type=network) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:07.639790 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d31a379-2816-4f02-86f7-b4d2925d09f6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:07.642132 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d31a379-2816-4f02-86f7-b4d2925d09f6 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:07.648026 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 415/824] 10.222.0.77 () {56 vars in 1112 bytes} [Tue Aug 30 05:00:07 2022] GET /identity/v3/services?type=network => generated 350 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:07.653006 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d31a379-2816-4f02-86f7-b4d2925d09f6 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:07.745351 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9dcc2f5-6c11-4490-a629-155bc5bfeb79 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:07.763185 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d31a379-2816-4f02-86f7-b4d2925d09f6 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:07.782250 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-951a74ff-503d-4228-ac4f-f2260899bb00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:07.783263 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-951a74ff-503d-4228-ac4f-f2260899bb00 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:07.783730 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-951a74ff-503d-4228-ac4f-f2260899bb00 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:07.784184 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-951a74ff-503d-4228-ac4f-f2260899bb00 None admin] PATH_INFO: `/v3/users` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:07.785217 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-951a74ff-503d-4228-ac4f-f2260899bb00 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:07.785707 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-951a74ff-503d-4228-ac4f-f2260899bb00 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:07.787355 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-951a74ff-503d-4228-ac4f-f2260899bb00 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:07.788226 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:07.800821 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9aa362a-ce13-4495-93fc-c21d2913baff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:07.801431 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9aa362a-ce13-4495-93fc-c21d2913baff None admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:07.801702 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9aa362a-ce13-4495-93fc-c21d2913baff None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:07.802045 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9aa362a-ce13-4495-93fc-c21d2913baff None admin] PATH_INFO: `/v3/endpoints` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:07.802775 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9aa362a-ce13-4495-93fc-c21d2913baff None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:07.804765 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9aa362a-ce13-4495-93fc-c21d2913baff None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:07.817577 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 410/825] 10.222.0.77 () {56 vars in 1133 bytes} [Tue Aug 30 05:00:07 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 05:00:07.823751 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-951a74ff-503d-4228-ac4f-f2260899bb00 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:07.851336 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 416/826] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:00:07 2022] POST /identity/v3/endpoints => generated 337 bytes in 198 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:00:07.876251 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:07.968408 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-951a74ff-503d-4228-ac4f-f2260899bb00 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:08.026486 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5584859-8dc3-44c3-89bb-6f845d88096e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:08.027125 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5584859-8dc3-44c3-89bb-6f845d88096e None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:08.027401 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5584859-8dc3-44c3-89bb-6f845d88096e None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:08.027668 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5584859-8dc3-44c3-89bb-6f845d88096e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:08.135003 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5584859-8dc3-44c3-89bb-6f845d88096e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:08.136394 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5584859-8dc3-44c3-89bb-6f845d88096e None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:08.263769 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2008503b-bd07-4fda-89b1-f762650ddaed None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:08.264137 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2008503b-bd07-4fda-89b1-f762650ddaed None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:08.264318 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2008503b-bd07-4fda-89b1-f762650ddaed None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:08.269812 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 417/827] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:00:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 05:00:08.271357 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c767b195-bdf5-4b4e-8442-bfc73d47c66b None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:08.271642 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c767b195-bdf5-4b4e-8442-bfc73d47c66b None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:08.271915 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c767b195-bdf5-4b4e-8442-bfc73d47c66b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:08.293597 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:08.298165 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-c767b195-bdf5-4b4e-8442-bfc73d47c66b None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:00:08.299415 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-c767b195-bdf5-4b4e-8442-bfc73d47c66b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:00:08.380916 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 411/828] 10.222.0.77 () {58 vars in 1282 bytes} [Tue Aug 30 05:00:07 2022] GET /identity/v3/auth/tokens => generated 1161 bytes in 557 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Aug 30 05:00:08.383555 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 418/829] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:00:08 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 30 05:00:08.386246 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5584859-8dc3-44c3-89bb-6f845d88096e None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:08.510790 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5584859-8dc3-44c3-89bb-6f845d88096e None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:08.537997 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:08.546786 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-993f3cea-2da2-4521-8542-528f8139219c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:08.547457 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-993f3cea-2da2-4521-8542-528f8139219c None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:08.548175 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-993f3cea-2da2-4521-8542-528f8139219c None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:08.548456 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-993f3cea-2da2-4521-8542-528f8139219c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:08.559519 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-993f3cea-2da2-4521-8542-528f8139219c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:08.560965 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-993f3cea-2da2-4521-8542-528f8139219c None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:08.575350 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-993f3cea-2da2-4521-8542-528f8139219c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 05:00:08.575350 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:08.575350 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:00:08.575350 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:00:08.575350 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:00:08.575350 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:00:08.575350 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:00:08.575350 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:00:08.575350 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:00:08.575350 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:00:08.575350 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:00:08.575350 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:00:08.575350 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 05:00:08.575350 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 05:00:08.575350 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 05:00:08.575350 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 05:00:08.576267 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:00:08.576267 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:00:08.576267 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:00:08.576267 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:00:08.576267 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:00:08.576267 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:00:08.576267 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:00:08.576267 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 05:00:08.576267 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:00:08.576267 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:00:08.576267 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:00:08.576267 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:00:08.576267 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:00:08.576267 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:00:08.576267 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:00:08.576267 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:00:08.576267 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 05:00:08.577644 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 05:00:08.577644 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 05:00:08.577644 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:00:08.577644 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 412/830] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 05:00:08 2022] GET /identity/v3/domains/Default => generated 87 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:00:08.582059 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c767b195-bdf5-4b4e-8442-bfc73d47c66b None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:08.687944 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c767b195-bdf5-4b4e-8442-bfc73d47c66b None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:08.698157 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:08.712646 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-993f3cea-2da2-4521-8542-528f8139219c None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:08.734690 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4877b550-0572-459b-859f-8018171a1283 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:08.735346 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4877b550-0572-459b-859f-8018171a1283 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:08.735629 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4877b550-0572-459b-859f-8018171a1283 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:08.735900 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4877b550-0572-459b-859f-8018171a1283 None admin] PATH_INFO: `/v3/domains` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:08.736775 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4877b550-0572-459b-859f-8018171a1283 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:08.737082 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4877b550-0572-459b-859f-8018171a1283 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:08.739001 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4877b550-0572-459b-859f-8018171a1283 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:08.755877 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 419/831] 10.222.0.77 () {56 vars in 1109 bytes} [Tue Aug 30 05:00:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:08.761418 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4877b550-0572-459b-859f-8018171a1283 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:08.848566 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-993f3cea-2da2-4521-8542-528f8139219c None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:08.896051 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4877b550-0572-459b-859f-8018171a1283 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:08.897052 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b92f887-08ec-408e-90e3-35caa3c89ba6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:08.898967 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b92f887-08ec-408e-90e3-35caa3c89ba6 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:08.900509 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b92f887-08ec-408e-90e3-35caa3c89ba6 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:08.901017 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b92f887-08ec-408e-90e3-35caa3c89ba6 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:08.915194 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b92f887-08ec-408e-90e3-35caa3c89ba6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 05:00:08.923842 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-1b92f887-08ec-408e-90e3-35caa3c89ba6 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 05:00:08.923842 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:08.923842 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:00:08.923842 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:00:08.923842 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:00:08.923842 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:00:08.923842 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:00:08.923842 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:00:08.923842 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:00:08.923842 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:00:08.923842 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:00:08.923842 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:00:08.923842 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 05:00:08.923842 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 05:00:08.923842 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 05:00:08.923842 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 05:00:08.924769 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 05:00:08.924769 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 05:00:08.924769 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 05:00:08.924769 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 05:00:08.924769 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:00:08.924769 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:00:08.924769 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:00:08.924769 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:00:08.924769 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:00:08.924769 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:00:08.924769 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:00:08.924769 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 05:00:08.924769 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:00:08.924769 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:00:08.924769 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:00:08.924769 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:00:08.925605 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:00:08.925605 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:00:08.925605 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:00:08.925605 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:00:08.925605 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 05:00:08.925605 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 05:00:08.925605 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 05:00:08.925605 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 05:00:08.925605 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 05:00:08.925605 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 05:00:08.925605 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 05:00:08.925605 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:00:08.928706 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 413/832] 10.222.0.77 () {56 vars in 1096 bytes} [Tue Aug 30 05:00:08 2022] GET /identity/v3/roles/admin => generated 83 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 05:00:08.934620 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b92f887-08ec-408e-90e3-35caa3c89ba6 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:08.941186 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc3ed227-56aa-45d5-a417-6205411041bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:08.941760 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc3ed227-56aa-45d5-a417-6205411041bc None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:08.942039 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc3ed227-56aa-45d5-a417-6205411041bc None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:08.942307 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc3ed227-56aa-45d5-a417-6205411041bc None admin] PATH_INFO: `/v3/projects/service` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:08.950878 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc3ed227-56aa-45d5-a417-6205411041bc None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:08.952922 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc3ed227-56aa-45d5-a417-6205411041bc None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:08.968282 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-cc3ed227-56aa-45d5-a417-6205411041bc None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 05:00:08.968282 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:08.968282 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:00:08.968282 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:00:08.968282 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:00:08.968282 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:00:08.968282 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:00:08.968282 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:00:08.968282 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:00:08.968282 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:00:08.968282 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:00:08.968282 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:00:08.968282 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 05:00:08.968282 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 05:00:08.968282 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 05:00:08.969200 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 05:00:08.969200 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:00:08.969200 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:00:08.969200 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:00:08.969200 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:00:08.969200 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:00:08.969200 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:00:08.969200 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:00:08.969200 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 05:00:08.969200 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:00:08.969200 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:00:08.969200 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:00:08.969200 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:00:08.969200 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:00:08.969200 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:00:08.969200 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:00:08.969200 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:00:08.970091 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 05:00:08.970091 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 05:00:08.970091 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 05:00:08.970091 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 05:00:08.970091 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 05:00:08.970091 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 05:00:08.970091 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 05:00:08.970091 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:00:08.970091 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 420/833] 10.222.0.77 () {56 vars in 1111 bytes} [Tue Aug 30 05:00:08 2022] GET /identity/v3/projects/service => generated 88 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:00:08.974803 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc3ed227-56aa-45d5-a417-6205411041bc None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:08.992106 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:09.017956 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:09.057821 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b92f887-08ec-408e-90e3-35caa3c89ba6 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:09.096492 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91f09a0d-fa03-4a5e-b25e-0a217d9edd2b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:09.097199 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91f09a0d-fa03-4a5e-b25e-0a217d9edd2b None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:09.097357 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91f09a0d-fa03-4a5e-b25e-0a217d9edd2b None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:09.097635 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91f09a0d-fa03-4a5e-b25e-0a217d9edd2b None admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:09.098297 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91f09a0d-fa03-4a5e-b25e-0a217d9edd2b None admin] RBAC: Adding query filter params (name=admin) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:09.098601 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91f09a0d-fa03-4a5e-b25e-0a217d9edd2b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:09.100005 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91f09a0d-fa03-4a5e-b25e-0a217d9edd2b None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:09.114473 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 414/834] 10.222.0.77 () {56 vars in 1099 bytes} [Tue Aug 30 05:00:08 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:09.115422 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91f09a0d-fa03-4a5e-b25e-0a217d9edd2b None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:09.119229 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc3ed227-56aa-45d5-a417-6205411041bc None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:09.187675 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db680a4a-faf7-4681-94bf-287bd96b3fed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:09.189303 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db680a4a-faf7-4681-94bf-287bd96b3fed None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:09.190068 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db680a4a-faf7-4681-94bf-287bd96b3fed None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:09.190858 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db680a4a-faf7-4681-94bf-287bd96b3fed None admin] PATH_INFO: `/v3/projects` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:09.192586 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db680a4a-faf7-4681-94bf-287bd96b3fed None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:09.193563 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db680a4a-faf7-4681-94bf-287bd96b3fed None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:09.197234 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db680a4a-faf7-4681-94bf-287bd96b3fed None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:09.220504 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 421/835] 10.222.0.77 () {56 vars in 1148 bytes} [Tue Aug 30 05:00:08 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 246 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 05:00:09.227452 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db680a4a-faf7-4681-94bf-287bd96b3fed None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:09.229239 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91f09a0d-fa03-4a5e-b25e-0a217d9edd2b None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:09.231109 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:09.278834 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c819072-9f01-4adc-8835-d0df89c70e28 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:09.279881 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c819072-9f01-4adc-8835-d0df89c70e28 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:09.280856 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c819072-9f01-4adc-8835-d0df89c70e28 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:09.281335 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c819072-9f01-4adc-8835-d0df89c70e28 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:09.294837 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c819072-9f01-4adc-8835-d0df89c70e28 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:09.296694 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c819072-9f01-4adc-8835-d0df89c70e28 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:09.311920 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-8c819072-9f01-4adc-8835-d0df89c70e28 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 05:00:09.311920 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:09.311920 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:00:09.311920 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:00:09.311920 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:00:09.311920 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:00:09.311920 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:00:09.311920 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:00:09.311920 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:00:09.311920 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:00:09.311920 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:00:09.311920 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:00:09.311920 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 05:00:09.311920 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 05:00:09.311920 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 05:00:09.311920 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 05:00:09.312861 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:00:09.312861 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:00:09.312861 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:00:09.312861 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:00:09.312861 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:00:09.312861 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:00:09.312861 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:00:09.312861 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 05:00:09.312861 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:00:09.312861 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:00:09.312861 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:00:09.312861 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:00:09.312861 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:00:09.312861 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:00:09.312861 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:00:09.312861 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:00:09.312861 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 05:00:09.313758 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 05:00:09.313758 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 05:00:09.313758 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:00:09.313758 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 415/836] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 05:00:09 2022] GET /identity/v3/domains/Default => generated 87 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:00:09.314419 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f65a71da-9f0b-4050-8689-42a300eda806 None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:09.314419 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f65a71da-9f0b-4050-8689-42a300eda806 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:09.314684 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f65a71da-9f0b-4050-8689-42a300eda806 None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:09.316041 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 416/837] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:00:09 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:00:09.318616 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f65a71da-9f0b-4050-8689-42a300eda806 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:09.338642 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:09.352513 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db680a4a-faf7-4681-94bf-287bd96b3fed None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:09.388495 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67703656-4fbe-4c07-a5f0-b456be5ac615 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:09.389088 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67703656-4fbe-4c07-a5f0-b456be5ac615 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:09.389708 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67703656-4fbe-4c07-a5f0-b456be5ac615 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:09.389708 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67703656-4fbe-4c07-a5f0-b456be5ac615 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:09.390329 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67703656-4fbe-4c07-a5f0-b456be5ac615 None admin] RBAC: Adding query filter params (role.id=9d9d5d2c270146488249b7ed943280db, scope.project.id=e9919141e3b947ab8d91b55bba4a7de6, user.id=886eb5e00bbf4ec292eb962b665b37d0) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:09.390635 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67703656-4fbe-4c07-a5f0-b456be5ac615 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:09.392082 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67703656-4fbe-4c07-a5f0-b456be5ac615 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:09.406843 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 422/838] 10.222.0.77 () {56 vars in 1374 bytes} [Tue Aug 30 05:00:09 2022] GET /identity/v3/role_assignments?user.id=886eb5e00bbf4ec292eb962b665b37d0&scope.project.id=e9919141e3b947ab8d91b55bba4a7de6&role.id=9d9d5d2c270146488249b7ed943280db => generated 608 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:09.406843 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3567c70c-9c96-4619-93cc-94da9494e4f7 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:09.406843 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3567c70c-9c96-4619-93cc-94da9494e4f7 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:09.406843 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3567c70c-9c96-4619-93cc-94da9494e4f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:09.427485 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-3567c70c-9c96-4619-93cc-94da9494e4f7 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:00:09.429861 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-3567c70c-9c96-4619-93cc-94da9494e4f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:00:09.441403 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f65a71da-9f0b-4050-8689-42a300eda806 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:09.494001 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03425801-1df6-463e-9031-7f7f27648ca0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:09.495096 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03425801-1df6-463e-9031-7f7f27648ca0 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:09.495608 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03425801-1df6-463e-9031-7f7f27648ca0 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:09.496064 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03425801-1df6-463e-9031-7f7f27648ca0 None admin] PATH_INFO: `/v3/domains` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:09.497033 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03425801-1df6-463e-9031-7f7f27648ca0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:09.497593 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03425801-1df6-463e-9031-7f7f27648ca0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:09.499709 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03425801-1df6-463e-9031-7f7f27648ca0 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:09.520617 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 417/839] 10.222.0.77 () {56 vars in 1109 bytes} [Tue Aug 30 05:00:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 05:00:09.521843 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 423/840] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:00:09 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 05:00:09.522105 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03425801-1df6-463e-9031-7f7f27648ca0 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:09.546504 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:09.638652 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03425801-1df6-463e-9031-7f7f27648ca0 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:09.661088 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-749793f8-8b52-4aa4-ae71-b4609418059b None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:09.661552 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-749793f8-8b52-4aa4-ae71-b4609418059b None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:09.661945 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-749793f8-8b52-4aa4-ae71-b4609418059b None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:09.664733 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 424/841] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:00:09 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 05:00:09.670164 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d0dee2d-892c-4d82-93c8-2257cd1ae772 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:09.670690 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d0dee2d-892c-4d82-93c8-2257cd1ae772 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:09.671208 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d0dee2d-892c-4d82-93c8-2257cd1ae772 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:09.676881 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f486a96c-ba30-4d2a-89e2-5483d33a020f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:09.677508 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f486a96c-ba30-4d2a-89e2-5483d33a020f None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:09.677785 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f486a96c-ba30-4d2a-89e2-5483d33a020f None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:09.678059 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f486a96c-ba30-4d2a-89e2-5483d33a020f None admin] PATH_INFO: `/v3/users/swift` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:09.687827 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f486a96c-ba30-4d2a-89e2-5483d33a020f None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:09.689275 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f486a96c-ba30-4d2a-89e2-5483d33a020f None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:09.691872 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-6d0dee2d-892c-4d82-93c8-2257cd1ae772 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:00:09.693264 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-6d0dee2d-892c-4d82-93c8-2257cd1ae772 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:00:09.702402 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-f486a96c-ba30-4d2a-89e2-5483d33a020f None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Aug 30 05:00:09.702402 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:09.702402 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:00:09.702402 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:00:09.702402 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:00:09.702402 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:00:09.702402 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:00:09.702402 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:00:09.702402 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:00:09.702402 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:00:09.702402 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:00:09.702402 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:00:09.702402 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 05:00:09.702402 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 05:00:09.702402 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 05:00:09.702402 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 05:00:09.703403 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:00:09.703403 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:00:09.703403 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 05:00:09.703403 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 05:00:09.703403 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 05:00:09.703403 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 05:00:09.703403 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:00:09.703403 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:00:09.703403 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:00:09.703403 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:00:09.703403 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:00:09.703403 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 05:00:09.703403 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:00:09.703403 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:00:09.703403 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:00:09.703403 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:00:09.703403 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:00:09.703403 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:00:09.704695 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:00:09.704695 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:00:09.704695 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 05:00:09.704695 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 05:00:09.704695 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 05:00:09.704695 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 05:00:09.704695 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 05:00:09.704695 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 05:00:09.704695 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Aug 30 05:00:09.704695 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:00:09.704695 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 418/842] 10.222.0.77 () {56 vars in 1096 bytes} [Tue Aug 30 05:00:09 2022] GET /identity/v3/users/swift => generated 83 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:00:09.709590 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f486a96c-ba30-4d2a-89e2-5483d33a020f None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:09.786568 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 425/843] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:00:09 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 30 05:00:09.788563 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d0dee2d-892c-4d82-93c8-2257cd1ae772 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:09.808519 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:09.814468 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:09.841910 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f486a96c-ba30-4d2a-89e2-5483d33a020f None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:09.888341 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e97ea51-ff75-4b46-9572-33bbb7a58c3d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:09.889040 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e97ea51-ff75-4b46-9572-33bbb7a58c3d None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:09.889389 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e97ea51-ff75-4b46-9572-33bbb7a58c3d None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:09.889725 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e97ea51-ff75-4b46-9572-33bbb7a58c3d None admin] PATH_INFO: `/v3/users` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:09.890968 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e97ea51-ff75-4b46-9572-33bbb7a58c3d None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:09.891341 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e97ea51-ff75-4b46-9572-33bbb7a58c3d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:09.893214 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e97ea51-ff75-4b46-9572-33bbb7a58c3d None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:09.917518 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 419/844] 10.222.0.77 () {56 vars in 1135 bytes} [Tue Aug 30 05:00:09 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 05:00:09.919913 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d0dee2d-892c-4d82-93c8-2257cd1ae772 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:09.922348 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e97ea51-ff75-4b46-9572-33bbb7a58c3d None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:09.964248 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a4a8d61-6767-4db0-acee-4c03e49a6096 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:09.965221 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a4a8d61-6767-4db0-acee-4c03e49a6096 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:09.965754 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a4a8d61-6767-4db0-acee-4c03e49a6096 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:09.966335 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a4a8d61-6767-4db0-acee-4c03e49a6096 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:09.977869 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a4a8d61-6767-4db0-acee-4c03e49a6096 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 05:00:09.983723 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-5a4a8d61-6767-4db0-acee-4c03e49a6096 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 30 05:00:09.983723 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:09.983723 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:00:09.983723 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:00:09.983723 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:00:09.983723 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:00:09.983723 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:00:09.983723 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:00:09.983723 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:00:09.983723 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:00:09.983723 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:00:09.983723 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:00:09.983723 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 05:00:09.983723 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 05:00:09.983723 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 05:00:09.983723 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 05:00:09.984641 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 05:00:09.984641 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 05:00:09.984641 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 05:00:09.984641 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 05:00:09.984641 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:00:09.984641 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:00:09.984641 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:00:09.984641 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:00:09.984641 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:00:09.984641 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:00:09.984641 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:00:09.984641 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 05:00:09.984641 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:00:09.984641 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:00:09.984641 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:00:09.984641 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:00:09.985486 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:00:09.985486 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:00:09.985486 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:00:09.985486 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:00:09.985486 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 05:00:09.985486 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 05:00:09.985486 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 05:00:09.985486 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 05:00:09.985486 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 05:00:09.985486 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 05:00:09.985486 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 30 05:00:09.985486 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:00:09.988135 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 426/845] 10.222.0.77 () {56 vars in 1102 bytes} [Tue Aug 30 05:00:09 2022] GET /identity/v3/roles/service => generated 85 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 05:00:09.994491 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a4a8d61-6767-4db0-acee-4c03e49a6096 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:10.048101 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e97ea51-ff75-4b46-9572-33bbb7a58c3d None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:10.074232 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:10.081100 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:10.088092 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-258aedb6-5edd-460a-ae9c-741f1c3063b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:10.088690 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-258aedb6-5edd-460a-ae9c-741f1c3063b4 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:10.088966 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-258aedb6-5edd-460a-ae9c-741f1c3063b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:10.089235 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-258aedb6-5edd-460a-ae9c-741f1c3063b4 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:10.110640 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a4a8d61-6767-4db0-acee-4c03e49a6096 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:10.151427 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8cc0942-3c4b-4fc1-819f-c8782902a20f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:10.152018 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8cc0942-3c4b-4fc1-819f-c8782902a20f None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:10.152298 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8cc0942-3c4b-4fc1-819f-c8782902a20f None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:10.152560 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8cc0942-3c4b-4fc1-819f-c8782902a20f None admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:10.153210 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8cc0942-3c4b-4fc1-819f-c8782902a20f None admin] RBAC: Adding query filter params (name=service) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:10.153501 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8cc0942-3c4b-4fc1-819f-c8782902a20f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:10.154867 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8cc0942-3c4b-4fc1-819f-c8782902a20f None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:10.160599 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-258aedb6-5edd-460a-ae9c-741f1c3063b4 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:10.162144 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-258aedb6-5edd-460a-ae9c-741f1c3063b4 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:10.166875 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 427/846] 10.222.0.77 () {56 vars in 1103 bytes} [Tue Aug 30 05:00:09 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:10.166875 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8cc0942-3c4b-4fc1-819f-c8782902a20f None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:10.271388 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 420/847] 10.222.0.77 () {58 vars in 1282 bytes} [Tue Aug 30 05:00:09 2022] GET /identity/v3/auth/tokens => generated 1161 bytes in 345 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Aug 30 05:00:10.271580 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8cc0942-3c4b-4fc1-819f-c8782902a20f None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:10.271672 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-258aedb6-5edd-460a-ae9c-741f1c3063b4 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:10.304998 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:10.307551 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15a4ee8c-b810-4790-8055-9be004cc8ce8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:10.308373 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15a4ee8c-b810-4790-8055-9be004cc8ce8 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:10.308650 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15a4ee8c-b810-4790-8055-9be004cc8ce8 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:10.309640 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15a4ee8c-b810-4790-8055-9be004cc8ce8 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:10.310223 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15a4ee8c-b810-4790-8055-9be004cc8ce8 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:10.311604 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15a4ee8c-b810-4790-8055-9be004cc8ce8 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:10.313509 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:10.324345 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-15a4ee8c-b810-4790-8055-9be004cc8ce8 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 30 05:00:10.324345 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:10.324345 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 30 05:00:10.324345 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 30 05:00:10.324345 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 30 05:00:10.324345 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 30 05:00:10.324345 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 30 05:00:10.324345 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 05:00:10.324345 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 30 05:00:10.324345 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:00:10.324345 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 05:00:10.324345 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:00:10.324345 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:10.324345 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:00:10.324345 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:00:10.324345 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:00:10.324345 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:00:10.325349 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:00:10.325349 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:00:10.325349 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:00:10.325349 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:00:10.325349 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:00:10.325349 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:00:10.325349 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 30 05:00:10.325349 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 30 05:00:10.325349 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 30 05:00:10.325349 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 30 05:00:10.325349 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:00:10.325349 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:00:10.325349 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:00:10.325349 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:00:10.325349 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:00:10.325349 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:00:10.326182 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:00:10.326182 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 05:00:10.326182 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:00:10.326182 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:00:10.326182 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:00:10.326182 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:00:10.326182 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:00:10.326182 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:00:10.326182 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:00:10.326182 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:00:10.326182 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 30 05:00:10.326182 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 05:00:10.326182 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 30 05:00:10.326182 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:00:10.330326 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 428/848] 10.222.0.77 () {56 vars in 1114 bytes} [Tue Aug 30 05:00:10 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:00:10.331145 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15a4ee8c-b810-4790-8055-9be004cc8ce8 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:10.389794 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-258aedb6-5edd-460a-ae9c-741f1c3063b4 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:10.432127 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04877739-398f-479c-ad4b-c3810d70f5d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:10.432807 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04877739-398f-479c-ad4b-c3810d70f5d6 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:10.433072 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04877739-398f-479c-ad4b-c3810d70f5d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:10.433342 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04877739-398f-479c-ad4b-c3810d70f5d6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:10.435802 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15a4ee8c-b810-4790-8055-9be004cc8ce8 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:10.446217 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04877739-398f-479c-ad4b-c3810d70f5d6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:10.448418 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04877739-398f-479c-ad4b-c3810d70f5d6 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:10.466367 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-04877739-398f-479c-ad4b-c3810d70f5d6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 05:00:10.466367 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:10.466367 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:00:10.466367 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:00:10.466367 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:00:10.466367 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:00:10.466367 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:00:10.466367 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:00:10.466367 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:00:10.466367 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:00:10.466367 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:00:10.466367 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:00:10.466367 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 05:00:10.466367 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 05:00:10.466367 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 05:00:10.466367 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 05:00:10.467514 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:00:10.467514 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:00:10.467514 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:00:10.467514 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:00:10.467514 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:00:10.467514 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:00:10.467514 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:00:10.467514 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 05:00:10.467514 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:00:10.467514 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:00:10.467514 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:00:10.467514 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:00:10.467514 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:00:10.467514 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:00:10.467514 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:00:10.467514 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:00:10.467514 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 05:00:10.468859 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 05:00:10.468859 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 05:00:10.468859 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:00:10.468859 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 421/849] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 05:00:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:00:10.469314 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04877739-398f-479c-ad4b-c3810d70f5d6 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:10.481992 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1cec33c-a616-4f21-bc59-a8284de33a35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:10.482722 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1cec33c-a616-4f21-bc59-a8284de33a35 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:10.483091 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1cec33c-a616-4f21-bc59-a8284de33a35 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:10.483493 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1cec33c-a616-4f21-bc59-a8284de33a35 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:10.495253 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1cec33c-a616-4f21-bc59-a8284de33a35 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:10.496806 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1cec33c-a616-4f21-bc59-a8284de33a35 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:10.511603 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-f1cec33c-a616-4f21-bc59-a8284de33a35 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 05:00:10.511603 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:10.511603 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:00:10.511603 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:00:10.511603 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:00:10.511603 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:00:10.511603 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:00:10.511603 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:00:10.511603 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:00:10.511603 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:00:10.511603 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:00:10.511603 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:00:10.511603 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 05:00:10.511603 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 05:00:10.511603 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 05:00:10.511603 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 05:00:10.512586 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:00:10.512586 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:00:10.512586 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:00:10.512586 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:00:10.512586 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:00:10.512586 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:00:10.512586 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:00:10.512586 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 05:00:10.512586 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:00:10.512586 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:00:10.512586 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:00:10.512586 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:00:10.512586 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:00:10.512586 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:00:10.512586 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:00:10.512586 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:00:10.512586 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 05:00:10.513531 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 05:00:10.513531 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 05:00:10.513531 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:00:10.513531 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 429/850] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 05:00:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:00:10.513531 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1cec33c-a616-4f21-bc59-a8284de33a35 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:10.564248 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:10.567440 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:10.583118 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04877739-398f-479c-ad4b-c3810d70f5d6 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:10.619057 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b7db9be-717c-4ea0-8d1f-5263b1982eb5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:10.619651 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b7db9be-717c-4ea0-8d1f-5263b1982eb5 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:10.619938 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b7db9be-717c-4ea0-8d1f-5263b1982eb5 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:10.620210 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b7db9be-717c-4ea0-8d1f-5263b1982eb5 None admin] PATH_INFO: `/v3/services` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:10.620861 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b7db9be-717c-4ea0-8d1f-5263b1982eb5 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:10.621163 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b7db9be-717c-4ea0-8d1f-5263b1982eb5 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:10.622605 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b7db9be-717c-4ea0-8d1f-5263b1982eb5 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:10.627519 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 422/851] 10.222.0.77 () {56 vars in 1114 bytes} [Tue Aug 30 05:00:10 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:10.628351 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b7db9be-717c-4ea0-8d1f-5263b1982eb5 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:10.637353 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1cec33c-a616-4f21-bc59-a8284de33a35 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:10.672460 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7133658c-946b-474d-a7b3-d675be5151fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:10.673054 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7133658c-946b-474d-a7b3-d675be5151fa None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:10.673334 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7133658c-946b-474d-a7b3-d675be5151fa None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:10.673606 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7133658c-946b-474d-a7b3-d675be5151fa None admin] PATH_INFO: `/v3/domains` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:10.674238 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7133658c-946b-474d-a7b3-d675be5151fa None admin] RBAC: Adding query filter params (name=Default) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:10.674536 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7133658c-946b-474d-a7b3-d675be5151fa None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:10.675980 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7133658c-946b-474d-a7b3-d675be5151fa None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:10.688175 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 430/852] 10.222.0.77 () {56 vars in 1109 bytes} [Tue Aug 30 05:00:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:10.689036 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7133658c-946b-474d-a7b3-d675be5151fa None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:10.737203 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b7db9be-717c-4ea0-8d1f-5263b1982eb5 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:10.771985 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32b08cde-4936-41f1-8eef-16c0d3f413d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:10.772766 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b08cde-4936-41f1-8eef-16c0d3f413d7 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:10.773118 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b08cde-4936-41f1-8eef-16c0d3f413d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:10.773461 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b08cde-4936-41f1-8eef-16c0d3f413d7 None admin] PATH_INFO: `/v3/domains` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:10.774170 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32b08cde-4936-41f1-8eef-16c0d3f413d7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:10.774544 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32b08cde-4936-41f1-8eef-16c0d3f413d7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:10.776005 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32b08cde-4936-41f1-8eef-16c0d3f413d7 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:10.792090 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 423/853] 10.222.0.77 () {56 vars in 1109 bytes} [Tue Aug 30 05:00:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:10.792090 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32b08cde-4936-41f1-8eef-16c0d3f413d7 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:10.800304 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:10.800802 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7133658c-946b-474d-a7b3-d675be5151fa None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:10.813864 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:10.842005 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25dadd67-4ff5-41a0-9c47-a1e398270e25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:10.842636 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25dadd67-4ff5-41a0-9c47-a1e398270e25 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:10.843018 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25dadd67-4ff5-41a0-9c47-a1e398270e25 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:10.843359 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25dadd67-4ff5-41a0-9c47-a1e398270e25 None admin] PATH_INFO: `/v3/services` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:10.844042 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25dadd67-4ff5-41a0-9c47-a1e398270e25 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:10.844443 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25dadd67-4ff5-41a0-9c47-a1e398270e25 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:10.845849 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25dadd67-4ff5-41a0-9c47-a1e398270e25 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:10.858861 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 431/854] 10.222.0.77 () {56 vars in 1114 bytes} [Tue Aug 30 05:00:10 2022] GET /identity/v3/services?type=volumev3 => generated 125 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:10.858861 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25dadd67-4ff5-41a0-9c47-a1e398270e25 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:10.923224 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32b08cde-4936-41f1-8eef-16c0d3f413d7 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:10.960346 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25dadd67-4ff5-41a0-9c47-a1e398270e25 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:10.970963 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ec78d04-0513-4cd1-bfd8-c33312300573 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:10.971639 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ec78d04-0513-4cd1-bfd8-c33312300573 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:10.971918 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ec78d04-0513-4cd1-bfd8-c33312300573 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:10.972186 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ec78d04-0513-4cd1-bfd8-c33312300573 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:10.984885 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ec78d04-0513-4cd1-bfd8-c33312300573 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:10.986608 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ec78d04-0513-4cd1-bfd8-c33312300573 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:11.001920 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed042e20-7578-4588-8df1-82e06df838d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:11.002712 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed042e20-7578-4588-8df1-82e06df838d5 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:11.003145 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed042e20-7578-4588-8df1-82e06df838d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:11.003478 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-2ec78d04-0513-4cd1-bfd8-c33312300573 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 30 05:00:11.003478 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:11.003478 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:00:11.003478 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:00:11.003478 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:00:11.003478 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:00:11.003478 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:00:11.003478 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:00:11.003478 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:00:11.003478 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:00:11.003478 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:00:11.003478 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:00:11.003478 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 05:00:11.003478 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 05:00:11.003478 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 05:00:11.003478 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 05:00:11.004570 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:00:11.004570 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:00:11.004570 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 05:00:11.004570 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 05:00:11.004570 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 05:00:11.004570 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 05:00:11.004570 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:00:11.004570 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:00:11.004570 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:00:11.004570 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:00:11.004570 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:00:11.004570 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 05:00:11.004570 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:00:11.004570 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:00:11.004570 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:00:11.004570 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:00:11.004570 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:00:11.004570 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:00:11.005707 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:00:11.005707 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:00:11.005707 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 05:00:11.005707 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 05:00:11.005707 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 05:00:11.005707 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 05:00:11.005707 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 05:00:11.005707 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 05:00:11.005707 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 30 05:00:11.005707 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:00:11.006374 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed042e20-7578-4588-8df1-82e06df838d5 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:11.007054 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 424/855] 10.222.0.77 () {56 vars in 1117 bytes} [Tue Aug 30 05:00:10 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 219 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:00:11.010258 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ec78d04-0513-4cd1-bfd8-c33312300573 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:11.017525 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed042e20-7578-4588-8df1-82e06df838d5 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:11.019140 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed042e20-7578-4588-8df1-82e06df838d5 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:11.031812 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-ed042e20-7578-4588-8df1-82e06df838d5 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 05:00:11.031812 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:11.031812 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:00:11.031812 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:00:11.031812 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:00:11.031812 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:00:11.031812 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:00:11.031812 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:00:11.031812 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:00:11.031812 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:00:11.031812 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:00:11.031812 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:00:11.031812 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 05:00:11.031812 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 05:00:11.031812 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 05:00:11.032948 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 05:00:11.032948 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:00:11.032948 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:00:11.032948 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:00:11.032948 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:00:11.032948 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:00:11.032948 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:00:11.032948 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:00:11.032948 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 05:00:11.032948 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:00:11.032948 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:00:11.032948 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:00:11.032948 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:00:11.032948 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:00:11.032948 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:00:11.032948 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:00:11.032948 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:00:11.033999 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 05:00:11.033999 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 05:00:11.033999 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 05:00:11.033999 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 05:00:11.033999 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 05:00:11.033999 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 05:00:11.033999 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 05:00:11.033999 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:00:11.033999 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 432/856] 10.222.0.77 () {56 vars in 1111 bytes} [Tue Aug 30 05:00:10 2022] GET /identity/v3/projects/service => generated 88 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:00:11.039263 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed042e20-7578-4588-8df1-82e06df838d5 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:11.085145 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:11.102599 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:11.140088 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ec78d04-0513-4cd1-bfd8-c33312300573 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:11.182116 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0c235f7-61ad-40e2-a831-3f72117f7975 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:11.182903 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0c235f7-61ad-40e2-a831-3f72117f7975 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:11.183283 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0c235f7-61ad-40e2-a831-3f72117f7975 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:11.183655 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0c235f7-61ad-40e2-a831-3f72117f7975 None admin] PATH_INFO: `/v3/users` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:11.184483 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0c235f7-61ad-40e2-a831-3f72117f7975 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:11.184891 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0c235f7-61ad-40e2-a831-3f72117f7975 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:11.185583 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed042e20-7578-4588-8df1-82e06df838d5 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:11.186527 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0c235f7-61ad-40e2-a831-3f72117f7975 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:11.209298 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 425/857] 10.222.0.77 () {56 vars in 1149 bytes} [Tue Aug 30 05:00:11 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 05:00:11.210871 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0c235f7-61ad-40e2-a831-3f72117f7975 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:11.232164 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b16e4354-83cc-4ad6-9996-9f1d49c269bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:11.233010 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b16e4354-83cc-4ad6-9996-9f1d49c269bd None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:11.233520 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b16e4354-83cc-4ad6-9996-9f1d49c269bd None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:11.234517 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b16e4354-83cc-4ad6-9996-9f1d49c269bd None admin] PATH_INFO: `/v3/projects` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:11.235829 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b16e4354-83cc-4ad6-9996-9f1d49c269bd None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:11.236285 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b16e4354-83cc-4ad6-9996-9f1d49c269bd None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:11.238914 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b16e4354-83cc-4ad6-9996-9f1d49c269bd None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:11.256585 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 433/858] 10.222.0.77 () {56 vars in 1148 bytes} [Tue Aug 30 05:00:11 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 05:00:11.266865 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b16e4354-83cc-4ad6-9996-9f1d49c269bd None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:11.321269 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0c235f7-61ad-40e2-a831-3f72117f7975 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:11.348288 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:11.358670 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7ec0dd5-d2e6-49a6-a987-77421eadeea9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:11.359214 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7ec0dd5-d2e6-49a6-a987-77421eadeea9 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:11.359491 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7ec0dd5-d2e6-49a6-a987-77421eadeea9 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:11.359760 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7ec0dd5-d2e6-49a6-a987-77421eadeea9 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:11.394406 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:11.401997 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b16e4354-83cc-4ad6-9996-9f1d49c269bd None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:11.436968 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7ec0dd5-d2e6-49a6-a987-77421eadeea9 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:11.438421 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7ec0dd5-d2e6-49a6-a987-77421eadeea9 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:11.438544 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2afa6e2-102a-4b76-ad13-38c2f0c73d0b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:11.439077 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2afa6e2-102a-4b76-ad13-38c2f0c73d0b None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:11.439357 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2afa6e2-102a-4b76-ad13-38c2f0c73d0b None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:11.439625 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2afa6e2-102a-4b76-ad13-38c2f0c73d0b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:11.440456 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2afa6e2-102a-4b76-ad13-38c2f0c73d0b None admin] RBAC: Adding query filter params (role.id=9d9d5d2c270146488249b7ed943280db, scope.project.id=e9919141e3b947ab8d91b55bba4a7de6, user.id=e0a7f7afc3d44ec788c05d005e52a1b8) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:11.440759 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2afa6e2-102a-4b76-ad13-38c2f0c73d0b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:11.442271 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2afa6e2-102a-4b76-ad13-38c2f0c73d0b None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:11.458961 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 434/859] 10.222.0.77 () {56 vars in 1374 bytes} [Tue Aug 30 05:00:11 2022] GET /identity/v3/role_assignments?user.id=e0a7f7afc3d44ec788c05d005e52a1b8&scope.project.id=e9919141e3b947ab8d91b55bba4a7de6&role.id=9d9d5d2c270146488249b7ed943280db => generated 608 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 05:00:11.461994 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f75cecd6-43b7-4631-b4de-a94f670ed4f0 None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:11.462834 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f75cecd6-43b7-4631-b4de-a94f670ed4f0 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:11.463313 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f75cecd6-43b7-4631-b4de-a94f670ed4f0 None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:11.465718 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 435/860] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:00:11 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:00:11.471728 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af0b2a83-6c1d-4f97-abf7-e6c40a5cd373 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:11.472947 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af0b2a83-6c1d-4f97-abf7-e6c40a5cd373 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:11.473436 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af0b2a83-6c1d-4f97-abf7-e6c40a5cd373 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:11.493902 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-af0b2a83-6c1d-4f97-abf7-e6c40a5cd373 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:00:11.495572 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-af0b2a83-6c1d-4f97-abf7-e6c40a5cd373 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:00:11.553331 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 426/861] 10.222.0.77 () {58 vars in 1282 bytes} [Tue Aug 30 05:00:11 2022] GET /identity/v3/auth/tokens => generated 1161 bytes in 343 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Aug 30 05:00:11.558953 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7ec0dd5-d2e6-49a6-a987-77421eadeea9 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:11.587320 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 436/862] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:00:11 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 05:00:11.615307 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:11.673612 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7ec0dd5-d2e6-49a6-a987-77421eadeea9 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:11.709074 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cfcfb6f-fcbf-4278-9260-f056ec1eece2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:11.709688 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cfcfb6f-fcbf-4278-9260-f056ec1eece2 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:11.709993 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cfcfb6f-fcbf-4278-9260-f056ec1eece2 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:11.710261 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cfcfb6f-fcbf-4278-9260-f056ec1eece2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:11.720584 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cfcfb6f-fcbf-4278-9260-f056ec1eece2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:11.722084 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cfcfb6f-fcbf-4278-9260-f056ec1eece2 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:11.736949 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-1cfcfb6f-fcbf-4278-9260-f056ec1eece2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 05:00:11.736949 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:11.736949 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:00:11.736949 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:00:11.736949 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:00:11.736949 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:00:11.736949 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:00:11.736949 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:00:11.736949 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:00:11.736949 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:00:11.736949 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:00:11.736949 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:00:11.736949 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 05:00:11.736949 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 05:00:11.736949 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 05:00:11.736949 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 05:00:11.737920 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:00:11.737920 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:00:11.737920 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:00:11.737920 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:00:11.737920 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:00:11.737920 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:00:11.737920 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:00:11.737920 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 05:00:11.737920 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:00:11.737920 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:00:11.737920 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:00:11.737920 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:00:11.737920 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:00:11.737920 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:00:11.737920 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:00:11.737920 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:00:11.737920 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 05:00:11.739460 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 05:00:11.739460 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 05:00:11.739460 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:00:11.739460 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 427/863] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 05:00:11 2022] GET /identity/v3/domains/Default => generated 87 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 05:00:11.744511 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af0b2a83-6c1d-4f97-abf7-e6c40a5cd373 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:11.837878 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:11.850523 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af0b2a83-6c1d-4f97-abf7-e6c40a5cd373 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:11.867920 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cfcfb6f-fcbf-4278-9260-f056ec1eece2 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:11.890739 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-854bd4fe-8640-46be-ab19-cd71801166a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:11.891381 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-854bd4fe-8640-46be-ab19-cd71801166a8 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:11.891656 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-854bd4fe-8640-46be-ab19-cd71801166a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:11.891928 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-854bd4fe-8640-46be-ab19-cd71801166a8 None admin] PATH_INFO: `/v3/domains` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:11.892868 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-854bd4fe-8640-46be-ab19-cd71801166a8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:11.893172 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-854bd4fe-8640-46be-ab19-cd71801166a8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:11.894594 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-854bd4fe-8640-46be-ab19-cd71801166a8 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:11.913925 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 437/864] 10.222.0.77 () {56 vars in 1109 bytes} [Tue Aug 30 05:00:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:11.915807 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-854bd4fe-8640-46be-ab19-cd71801166a8 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:11.966407 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:11.991686 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cfcfb6f-fcbf-4278-9260-f056ec1eece2 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:12.028256 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f00cdb94-ce12-4d75-9430-50577cd50554 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:12.028935 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f00cdb94-ce12-4d75-9430-50577cd50554 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:12.029211 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f00cdb94-ce12-4d75-9430-50577cd50554 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:12.029532 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f00cdb94-ce12-4d75-9430-50577cd50554 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:12.030113 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f00cdb94-ce12-4d75-9430-50577cd50554 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:12.031523 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f00cdb94-ce12-4d75-9430-50577cd50554 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:12.041642 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-f00cdb94-ce12-4d75-9430-50577cd50554 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 30 05:00:12.041642 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:12.041642 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 30 05:00:12.041642 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 30 05:00:12.041642 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 30 05:00:12.041642 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 30 05:00:12.041642 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 30 05:00:12.041642 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 05:00:12.041642 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 30 05:00:12.041642 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:00:12.041642 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 05:00:12.041642 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:00:12.041642 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:12.041642 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:00:12.041642 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:00:12.041642 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:00:12.042662 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:00:12.042662 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:00:12.042662 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:00:12.042662 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:00:12.042662 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:00:12.042662 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:00:12.042662 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:00:12.042662 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 30 05:00:12.042662 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 30 05:00:12.042662 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 30 05:00:12.042662 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 30 05:00:12.042662 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:00:12.042662 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:00:12.042662 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:00:12.042662 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:00:12.042662 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:00:12.042662 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:00:12.049086 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:00:12.049086 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 05:00:12.049086 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:00:12.049086 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:00:12.049086 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:00:12.049086 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:00:12.049086 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:00:12.049086 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:00:12.049086 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:00:12.049086 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:00:12.049086 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 30 05:00:12.049086 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 05:00:12.049086 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 30 05:00:12.049086 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:00:12.049086 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 428/865] 10.222.0.77 () {56 vars in 1132 bytes} [Tue Aug 30 05:00:11 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:00:12.052707 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f00cdb94-ce12-4d75-9430-50577cd50554 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:12.061814 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-854bd4fe-8640-46be-ab19-cd71801166a8 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:12.112893 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01c12449-b95a-4dc2-95e5-b874655a9cf5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:12.114124 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01c12449-b95a-4dc2-95e5-b874655a9cf5 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:12.114879 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01c12449-b95a-4dc2-95e5-b874655a9cf5 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:12.115384 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01c12449-b95a-4dc2-95e5-b874655a9cf5 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:12.125625 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01c12449-b95a-4dc2-95e5-b874655a9cf5 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:12.128065 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01c12449-b95a-4dc2-95e5-b874655a9cf5 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:12.142782 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-01c12449-b95a-4dc2-95e5-b874655a9cf5 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 05:00:12.142782 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:12.142782 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:00:12.142782 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:00:12.142782 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:00:12.142782 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:00:12.142782 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:00:12.142782 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:00:12.142782 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:00:12.142782 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:00:12.142782 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:00:12.142782 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:00:12.142782 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 05:00:12.142782 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 05:00:12.142782 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 05:00:12.143657 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 05:00:12.143657 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:00:12.143657 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:00:12.143657 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:00:12.143657 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:00:12.143657 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:00:12.143657 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:00:12.143657 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:00:12.143657 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 05:00:12.143657 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:00:12.143657 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:00:12.143657 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:00:12.143657 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:00:12.143657 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:00:12.143657 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:00:12.143657 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:00:12.143657 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:00:12.144538 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 05:00:12.144538 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 05:00:12.144538 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 05:00:12.144538 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 05:00:12.144538 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 05:00:12.144538 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 05:00:12.144538 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 05:00:12.144538 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:00:12.151478 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 438/866] 10.222.0.77 () {56 vars in 1111 bytes} [Tue Aug 30 05:00:11 2022] GET /identity/v3/projects/service => generated 88 bytes in 236 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:00:12.152535 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01c12449-b95a-4dc2-95e5-b874655a9cf5 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:12.169770 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:12.170542 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f00cdb94-ce12-4d75-9430-50577cd50554 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:12.212499 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0776210-0ebd-4ec5-91b4-eb59c979be9d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:12.213552 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0776210-0ebd-4ec5-91b4-eb59c979be9d None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:12.213552 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0776210-0ebd-4ec5-91b4-eb59c979be9d None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:12.214044 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0776210-0ebd-4ec5-91b4-eb59c979be9d None admin] PATH_INFO: `/v3/services` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:12.214839 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0776210-0ebd-4ec5-91b4-eb59c979be9d None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:12.215249 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0776210-0ebd-4ec5-91b4-eb59c979be9d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:12.216759 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0776210-0ebd-4ec5-91b4-eb59c979be9d None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:12.226692 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 429/867] 10.222.0.77 () {56 vars in 1126 bytes} [Tue Aug 30 05:00:12 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:12.227669 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0776210-0ebd-4ec5-91b4-eb59c979be9d None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:12.240295 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:12.286073 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01c12449-b95a-4dc2-95e5-b874655a9cf5 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:12.322271 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c564b3fa-205c-4fd2-ae3e-d4c505ccd12e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:12.323048 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c564b3fa-205c-4fd2-ae3e-d4c505ccd12e None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:12.323422 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c564b3fa-205c-4fd2-ae3e-d4c505ccd12e None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:12.323789 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c564b3fa-205c-4fd2-ae3e-d4c505ccd12e None admin] PATH_INFO: `/v3/projects` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:12.324570 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c564b3fa-205c-4fd2-ae3e-d4c505ccd12e None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:12.325235 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c564b3fa-205c-4fd2-ae3e-d4c505ccd12e None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:12.326589 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c564b3fa-205c-4fd2-ae3e-d4c505ccd12e None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:12.344518 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 439/868] 10.222.0.77 () {56 vars in 1148 bytes} [Tue Aug 30 05:00:12 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:12.345549 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c564b3fa-205c-4fd2-ae3e-d4c505ccd12e None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:12.371953 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0776210-0ebd-4ec5-91b4-eb59c979be9d None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:12.416311 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24b0aad7-8e0b-43c5-8ca2-f81fb318ecc2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:12.416847 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:12.417305 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24b0aad7-8e0b-43c5-8ca2-f81fb318ecc2 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:12.418197 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24b0aad7-8e0b-43c5-8ca2-f81fb318ecc2 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:12.418622 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24b0aad7-8e0b-43c5-8ca2-f81fb318ecc2 None admin] PATH_INFO: `/v3/services` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:12.419581 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24b0aad7-8e0b-43c5-8ca2-f81fb318ecc2 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:12.420177 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24b0aad7-8e0b-43c5-8ca2-f81fb318ecc2 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:12.421972 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24b0aad7-8e0b-43c5-8ca2-f81fb318ecc2 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:12.435480 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 430/869] 10.222.0.77 () {56 vars in 1126 bytes} [Tue Aug 30 05:00:12 2022] GET /identity/v3/services?type=compute_legacy => generated 131 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:12.464539 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c564b3fa-205c-4fd2-ae3e-d4c505ccd12e None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:12.503876 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7cfb1ca-7869-40d1-af85-5e3a8995ac0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:12.504497 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7cfb1ca-7869-40d1-af85-5e3a8995ac0d None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:12.504774 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7cfb1ca-7869-40d1-af85-5e3a8995ac0d None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:12.505043 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7cfb1ca-7869-40d1-af85-5e3a8995ac0d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:12.505776 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7cfb1ca-7869-40d1-af85-5e3a8995ac0d None admin] RBAC: Adding query filter params (role.id=821647e8b40542499667e29e1b301cb2, scope.project.id=e9919141e3b947ab8d91b55bba4a7de6, user.id=fe51dbec91624ed1ab00b441e29a3dd5) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:12.506081 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7cfb1ca-7869-40d1-af85-5e3a8995ac0d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:12.507630 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7cfb1ca-7869-40d1-af85-5e3a8995ac0d None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:12.515113 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 440/870] 10.222.0.77 () {56 vars in 1374 bytes} [Tue Aug 30 05:00:12 2022] GET /identity/v3/role_assignments?user.id=fe51dbec91624ed1ab00b441e29a3dd5&scope.project.id=e9919141e3b947ab8d91b55bba4a7de6&role.id=821647e8b40542499667e29e1b301cb2 => generated 608 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:12.899976 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-019621f7-a0ba-4d7a-8623-c60cd01d4e5d None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:12.900286 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-019621f7-a0ba-4d7a-8623-c60cd01d4e5d None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:12.900551 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-019621f7-a0ba-4d7a-8623-c60cd01d4e5d None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:12.902097 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 431/871] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:00:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:00:12.907660 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a41af99-f9b2-41bd-95e4-25de77df0e39 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:12.907945 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a41af99-f9b2-41bd-95e4-25de77df0e39 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:12.908216 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a41af99-f9b2-41bd-95e4-25de77df0e39 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:12.926566 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-8a41af99-f9b2-41bd-95e4-25de77df0e39 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:00:12.927809 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-8a41af99-f9b2-41bd-95e4-25de77df0e39 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:00:12.999513 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 441/872] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:00:12 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 05:00:13.269361 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-019621f7-a0ba-4d7a-8623-c60cd01d4e5d None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:13.341517 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:13.375799 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-019621f7-a0ba-4d7a-8623-c60cd01d4e5d None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:13.411003 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60b2dd42-67d4-423d-b0c0-351a6d6b7c4c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:13.411923 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b2dd42-67d4-423d-b0c0-351a6d6b7c4c None admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:13.412418 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b2dd42-67d4-423d-b0c0-351a6d6b7c4c None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:13.412925 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b2dd42-67d4-423d-b0c0-351a6d6b7c4c None admin] PATH_INFO: `/v3/services` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:13.413771 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60b2dd42-67d4-423d-b0c0-351a6d6b7c4c None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:13.415413 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60b2dd42-67d4-423d-b0c0-351a6d6b7c4c None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:13.433097 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 432/873] 10.222.0.77 () {60 vars in 1141 bytes} [Tue Aug 30 05:00:13 2022] POST /identity/v3/services => generated 251 bytes in 164 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:00:13.490369 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fba6bcb-93a9-4e79-903e-e46de14560f4 None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:13.491525 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fba6bcb-93a9-4e79-903e-e46de14560f4 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:13.491525 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fba6bcb-93a9-4e79-903e-e46de14560f4 None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:13.493066 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 442/874] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:00:13 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:00:13.498808 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b72be989-3c6d-47f7-94fc-78891b5a1023 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:13.499407 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b72be989-3c6d-47f7-94fc-78891b5a1023 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:13.499883 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b72be989-3c6d-47f7-94fc-78891b5a1023 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:13.523745 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-b72be989-3c6d-47f7-94fc-78891b5a1023 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:00:13.524884 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-b72be989-3c6d-47f7-94fc-78891b5a1023 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:00:13.611496 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 433/875] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:00:13 2022] POST /identity/v3/auth/tokens => generated 1262 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 05:00:13.878608 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fba6bcb-93a9-4e79-903e-e46de14560f4 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:13.944570 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:13.988664 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fba6bcb-93a9-4e79-903e-e46de14560f4 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:14.024546 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d56ff6a-2bab-444f-ae1c-bf2edd4e36fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:14.025170 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d56ff6a-2bab-444f-ae1c-bf2edd4e36fa None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:14.025444 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d56ff6a-2bab-444f-ae1c-bf2edd4e36fa None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:14.025706 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d56ff6a-2bab-444f-ae1c-bf2edd4e36fa None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:14.026277 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d56ff6a-2bab-444f-ae1c-bf2edd4e36fa None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:14.027686 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d56ff6a-2bab-444f-ae1c-bf2edd4e36fa None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:14.036798 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-3d56ff6a-2bab-444f-ae1c-bf2edd4e36fa None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 30 05:00:14.036798 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:14.036798 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 30 05:00:14.036798 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 30 05:00:14.036798 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 30 05:00:14.036798 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 30 05:00:14.036798 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 30 05:00:14.036798 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 05:00:14.036798 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 30 05:00:14.036798 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:00:14.036798 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 05:00:14.036798 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:00:14.036798 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:14.036798 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:00:14.036798 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:00:14.036798 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:00:14.036798 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:00:14.037842 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:00:14.037842 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:00:14.037842 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:00:14.037842 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:00:14.037842 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:00:14.037842 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:00:14.037842 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 30 05:00:14.037842 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 30 05:00:14.037842 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 30 05:00:14.037842 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 30 05:00:14.037842 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:00:14.037842 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:00:14.037842 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:00:14.037842 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:00:14.037842 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:00:14.037842 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:00:14.039225 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:00:14.039225 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 05:00:14.039225 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:00:14.039225 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:00:14.039225 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:00:14.039225 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:00:14.039225 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:00:14.039225 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:00:14.039225 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:00:14.039225 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:00:14.039225 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 30 05:00:14.039225 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 05:00:14.039225 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 30 05:00:14.039225 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:00:14.039225 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 443/876] 10.222.0.77 () {56 vars in 1126 bytes} [Tue Aug 30 05:00:13 2022] GET /identity/v3/services/object-store => generated 93 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:00:14.044622 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b72be989-3c6d-47f7-94fc-78891b5a1023 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:14.105490 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:14.156390 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b72be989-3c6d-47f7-94fc-78891b5a1023 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:14.192023 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f0f2373-03d4-4b22-9c4e-a8a74a1e89b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:14.192632 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f0f2373-03d4-4b22-9c4e-a8a74a1e89b8 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:14.192911 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f0f2373-03d4-4b22-9c4e-a8a74a1e89b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:14.193181 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f0f2373-03d4-4b22-9c4e-a8a74a1e89b8 None admin] PATH_INFO: `/v3/services` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:14.193809 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f0f2373-03d4-4b22-9c4e-a8a74a1e89b8 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:14.194137 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f0f2373-03d4-4b22-9c4e-a8a74a1e89b8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:14.195596 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f0f2373-03d4-4b22-9c4e-a8a74a1e89b8 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:14.200685 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 434/877] 10.222.0.77 () {56 vars in 1122 bytes} [Tue Aug 30 05:00:14 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:14.205966 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d56ff6a-2bab-444f-ae1c-bf2edd4e36fa None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:14.313002 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d56ff6a-2bab-444f-ae1c-bf2edd4e36fa None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:14.339836 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:14.348613 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8477a601-c1d7-4b77-8600-5fb8cc0a53d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:14.349217 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8477a601-c1d7-4b77-8600-5fb8cc0a53d3 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:14.349490 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8477a601-c1d7-4b77-8600-5fb8cc0a53d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:14.349754 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8477a601-c1d7-4b77-8600-5fb8cc0a53d3 None admin] PATH_INFO: `/v3/services` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:14.350549 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8477a601-c1d7-4b77-8600-5fb8cc0a53d3 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:14.351243 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8477a601-c1d7-4b77-8600-5fb8cc0a53d3 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:14.352783 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8477a601-c1d7-4b77-8600-5fb8cc0a53d3 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:14.358036 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 444/878] 10.222.0.77 () {56 vars in 1122 bytes} [Tue Aug 30 05:00:14 2022] GET /identity/v3/services?type=object-store => generated 129 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:14.481804 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d4fadd-577e-4738-a798-c511606a37a1 None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:14.482792 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d4fadd-577e-4738-a798-c511606a37a1 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:14.483312 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d4fadd-577e-4738-a798-c511606a37a1 None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:14.485229 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 435/879] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:00:14 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:00:14.491130 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8579810-79d9-49c0-ba10-e5345bc940da None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:14.492028 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8579810-79d9-49c0-ba10-e5345bc940da None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:14.492527 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8579810-79d9-49c0-ba10-e5345bc940da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:14.511960 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-e8579810-79d9-49c0-ba10-e5345bc940da None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:00:14.513484 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-e8579810-79d9-49c0-ba10-e5345bc940da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:00:14.538803 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a25984a9-ed59-47ae-a696-d0c406ec8606 None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:14.539125 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a25984a9-ed59-47ae-a696-d0c406ec8606 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:14.539389 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a25984a9-ed59-47ae-a696-d0c406ec8606 None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:14.540953 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 436/880] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:00:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:00:14.545825 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d3eb6ae-dfa4-4b0f-9183-46fce94f65c6 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:14.546107 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d3eb6ae-dfa4-4b0f-9183-46fce94f65c6 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:14.546385 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d3eb6ae-dfa4-4b0f-9183-46fce94f65c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:14.564775 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-8d3eb6ae-dfa4-4b0f-9183-46fce94f65c6 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:00:14.566004 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-8d3eb6ae-dfa4-4b0f-9183-46fce94f65c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:00:14.598618 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 445/881] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:00:14 2022] POST /identity/v3/auth/tokens => generated 1262 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 05:00:14.646859 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 437/882] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:00:14 2022] POST /identity/v3/auth/tokens => generated 1262 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 30 05:00:14.874402 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8579810-79d9-49c0-ba10-e5345bc940da None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:14.921473 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d3eb6ae-dfa4-4b0f-9183-46fce94f65c6 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:14.987240 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:14.990963 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8579810-79d9-49c0-ba10-e5345bc940da None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:15.016612 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:15.029267 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d3eb6ae-dfa4-4b0f-9183-46fce94f65c6 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:15.034674 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ddc8af9-312d-49b3-a160-0a98cc37222d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:15.035310 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ddc8af9-312d-49b3-a160-0a98cc37222d None admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:15.035585 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ddc8af9-312d-49b3-a160-0a98cc37222d None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:15.035851 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ddc8af9-312d-49b3-a160-0a98cc37222d None admin] PATH_INFO: `/v3/services` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:15.036507 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ddc8af9-312d-49b3-a160-0a98cc37222d None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:15.037930 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ddc8af9-312d-49b3-a160-0a98cc37222d None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:15.063212 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 446/883] 10.222.0.77 () {60 vars in 1141 bytes} [Tue Aug 30 05:00:14 2022] POST /identity/v3/services => generated 269 bytes in 187 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:00:15.082893 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73cc7487-bc0d-4e8c-907e-7c0e35649d13 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:15.083964 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73cc7487-bc0d-4e8c-907e-7c0e35649d13 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:15.084464 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73cc7487-bc0d-4e8c-907e-7c0e35649d13 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:15.085502 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73cc7487-bc0d-4e8c-907e-7c0e35649d13 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:15.098087 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73cc7487-bc0d-4e8c-907e-7c0e35649d13 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 05:00:15.103748 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-73cc7487-bc0d-4e8c-907e-7c0e35649d13 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 30 05:00:15.103748 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:15.103748 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:00:15.103748 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:00:15.103748 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:00:15.103748 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:00:15.103748 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:00:15.103748 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:00:15.103748 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:00:15.103748 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:00:15.103748 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:00:15.103748 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:00:15.103748 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 05:00:15.103748 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 05:00:15.103748 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 05:00:15.103748 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 05:00:15.104684 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 05:00:15.104684 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 05:00:15.104684 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 05:00:15.104684 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 05:00:15.104684 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:00:15.104684 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:00:15.104684 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:00:15.104684 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:00:15.104684 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:00:15.104684 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:00:15.104684 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:00:15.104684 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 05:00:15.104684 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:00:15.104684 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:00:15.104684 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:00:15.104684 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:00:15.105544 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:00:15.105544 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:00:15.105544 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:00:15.105544 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:00:15.105544 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 05:00:15.105544 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 05:00:15.105544 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 05:00:15.105544 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 05:00:15.105544 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 05:00:15.105544 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 05:00:15.105544 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 30 05:00:15.105544 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:00:15.108387 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 438/884] 10.222.0.77 () {56 vars in 1120 bytes} [Tue Aug 30 05:00:14 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 05:00:15.114763 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ddc8af9-312d-49b3-a160-0a98cc37222d None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:15.232648 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ddc8af9-312d-49b3-a160-0a98cc37222d None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:15.271848 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47abd4d1-87e8-4c3c-b4d1-4f54a762030e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:15.272976 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47abd4d1-87e8-4c3c-b4d1-4f54a762030e None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:15.273249 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47abd4d1-87e8-4c3c-b4d1-4f54a762030e None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:15.273515 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47abd4d1-87e8-4c3c-b4d1-4f54a762030e None admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:15.274164 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47abd4d1-87e8-4c3c-b4d1-4f54a762030e None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:15.274459 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47abd4d1-87e8-4c3c-b4d1-4f54a762030e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:15.276049 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47abd4d1-87e8-4c3c-b4d1-4f54a762030e None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:15.284583 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 447/885] 10.222.0.77 () {56 vars in 1115 bytes} [Tue Aug 30 05:00:15 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:15.290187 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73cc7487-bc0d-4e8c-907e-7c0e35649d13 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:15.393084 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73cc7487-bc0d-4e8c-907e-7c0e35649d13 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:15.401125 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:15.428303 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8796194-8ab3-47f4-abb8-9ea381c5f8cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:15.429451 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8796194-8ab3-47f4-abb8-9ea381c5f8cb None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:15.429725 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8796194-8ab3-47f4-abb8-9ea381c5f8cb None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:15.429984 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8796194-8ab3-47f4-abb8-9ea381c5f8cb None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:15.440465 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8796194-8ab3-47f4-abb8-9ea381c5f8cb None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:15.441928 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8796194-8ab3-47f4-abb8-9ea381c5f8cb None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:15.444503 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc9e2b4d-a791-4859-be76-e21c6aca1ad1 None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:15.444779 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc9e2b4d-a791-4859-be76-e21c6aca1ad1 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:15.445053 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc9e2b4d-a791-4859-be76-e21c6aca1ad1 None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:15.450965 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 448/886] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:00:15 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 05:00:15.457620 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-b8796194-8ab3-47f4-abb8-9ea381c5f8cb None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 05:00:15.457620 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:15.457620 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:00:15.457620 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:00:15.457620 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:00:15.457620 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:00:15.457620 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:00:15.457620 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:00:15.457620 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:00:15.457620 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:00:15.457620 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:00:15.457620 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:00:15.457620 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 05:00:15.457620 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 05:00:15.457620 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 05:00:15.457620 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 05:00:15.458616 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:00:15.458616 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:00:15.458616 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:00:15.458616 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:00:15.458616 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:00:15.458616 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:00:15.458616 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:00:15.458616 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 05:00:15.458616 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:00:15.458616 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:00:15.458616 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:00:15.458616 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:00:15.458616 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:00:15.458616 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:00:15.458616 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:00:15.458616 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:00:15.458616 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 05:00:15.459587 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 05:00:15.459587 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 05:00:15.459587 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:00:15.459806 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a84b42d-cfea-4f6e-8c9f-43483d4a0874 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:15.459806 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a84b42d-cfea-4f6e-8c9f-43483d4a0874 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:15.459806 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a84b42d-cfea-4f6e-8c9f-43483d4a0874 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:15.463257 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 439/887] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 05:00:15 2022] GET /identity/v3/domains/Default => generated 87 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:00:15.465100 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8796194-8ab3-47f4-abb8-9ea381c5f8cb None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:15.484956 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-6a84b42d-cfea-4f6e-8c9f-43483d4a0874 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:00:15.486781 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-6a84b42d-cfea-4f6e-8c9f-43483d4a0874 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:00:15.534926 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:15.575023 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8796194-8ab3-47f4-abb8-9ea381c5f8cb None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:15.586990 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 449/888] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:00:15 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 05:00:15.614319 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bf0c1b9-c7ce-4aa2-b2d5-3294ed013c9e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:15.615288 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf0c1b9-c7ce-4aa2-b2d5-3294ed013c9e None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:15.615744 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf0c1b9-c7ce-4aa2-b2d5-3294ed013c9e None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:15.616703 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf0c1b9-c7ce-4aa2-b2d5-3294ed013c9e None admin] PATH_INFO: `/v3/domains` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:15.617505 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bf0c1b9-c7ce-4aa2-b2d5-3294ed013c9e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:15.617996 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bf0c1b9-c7ce-4aa2-b2d5-3294ed013c9e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:15.619596 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bf0c1b9-c7ce-4aa2-b2d5-3294ed013c9e None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:15.633526 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 440/889] 10.222.0.77 () {56 vars in 1109 bytes} [Tue Aug 30 05:00:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 05:00:15.639312 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a84b42d-cfea-4f6e-8c9f-43483d4a0874 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:15.740521 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a84b42d-cfea-4f6e-8c9f-43483d4a0874 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:15.775834 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bf23654-c858-47be-9869-dfdbea60feb0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:15.776453 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bf23654-c858-47be-9869-dfdbea60feb0 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:15.776726 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bf23654-c858-47be-9869-dfdbea60feb0 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:15.776995 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bf23654-c858-47be-9869-dfdbea60feb0 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:15.784203 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bf23654-c858-47be-9869-dfdbea60feb0 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:15.785636 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bf23654-c858-47be-9869-dfdbea60feb0 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:15.797695 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-8bf23654-c858-47be-9869-dfdbea60feb0 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 30 05:00:15.797695 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:15.797695 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:00:15.797695 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:00:15.797695 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:00:15.797695 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:00:15.797695 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:00:15.797695 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:00:15.797695 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:00:15.797695 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:00:15.797695 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:00:15.797695 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:00:15.797695 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 05:00:15.797695 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 05:00:15.797695 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 05:00:15.797695 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 05:00:15.798642 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:00:15.798642 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:00:15.798642 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 05:00:15.798642 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 05:00:15.798642 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 05:00:15.798642 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 05:00:15.798642 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:00:15.798642 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:00:15.798642 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:00:15.798642 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:00:15.798642 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:00:15.798642 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 05:00:15.798642 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:00:15.798642 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:00:15.798642 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:00:15.798642 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:00:15.798642 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:00:15.798642 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:00:15.799611 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:00:15.799611 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:00:15.799611 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 05:00:15.799611 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 05:00:15.799611 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 05:00:15.799611 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 05:00:15.799611 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 05:00:15.799611 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 05:00:15.799611 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 30 05:00:15.799611 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:00:15.799611 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 450/890] 10.222.0.77 () {56 vars in 1117 bytes} [Tue Aug 30 05:00:15 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:00:15.807076 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bf0c1b9-c7ce-4aa2-b2d5-3294ed013c9e None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:15.827210 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:15.871083 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bf23654-c858-47be-9869-dfdbea60feb0 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:15.906018 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:15.917861 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bf0c1b9-c7ce-4aa2-b2d5-3294ed013c9e None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:15.953434 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fca6c982-10be-4f86-9bfc-fbb42f25b31c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:15.954033 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fca6c982-10be-4f86-9bfc-fbb42f25b31c None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:15.954309 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fca6c982-10be-4f86-9bfc-fbb42f25b31c None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:15.954579 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fca6c982-10be-4f86-9bfc-fbb42f25b31c None admin] PATH_INFO: `/v3/users` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:15.955331 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fca6c982-10be-4f86-9bfc-fbb42f25b31c None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:15.955635 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fca6c982-10be-4f86-9bfc-fbb42f25b31c None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:15.957057 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fca6c982-10be-4f86-9bfc-fbb42f25b31c None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:15.974858 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 441/891] 10.222.0.77 () {56 vars in 1149 bytes} [Tue Aug 30 05:00:15 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:15.980673 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fca6c982-10be-4f86-9bfc-fbb42f25b31c None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:15.993811 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bf23654-c858-47be-9869-dfdbea60feb0 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:16.030847 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6763f973-67e6-4de2-aeb8-e798d67a61c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:16.031687 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6763f973-67e6-4de2-aeb8-e798d67a61c6 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:16.032025 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6763f973-67e6-4de2-aeb8-e798d67a61c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:16.032368 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6763f973-67e6-4de2-aeb8-e798d67a61c6 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:16.033135 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6763f973-67e6-4de2-aeb8-e798d67a61c6 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:16.034549 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6763f973-67e6-4de2-aeb8-e798d67a61c6 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:16.044503 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-6763f973-67e6-4de2-aeb8-e798d67a61c6 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 30 05:00:16.044503 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:16.044503 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 30 05:00:16.044503 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 30 05:00:16.044503 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 30 05:00:16.044503 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 30 05:00:16.044503 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 30 05:00:16.044503 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 05:00:16.044503 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 30 05:00:16.044503 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:00:16.044503 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 05:00:16.044503 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:00:16.044503 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:16.044503 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:00:16.044503 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:00:16.044503 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:00:16.044503 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:00:16.045508 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:00:16.045508 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:00:16.045508 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:00:16.045508 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:00:16.045508 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:00:16.045508 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:00:16.045508 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 30 05:00:16.045508 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 30 05:00:16.045508 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 30 05:00:16.045508 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 30 05:00:16.045508 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:00:16.045508 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:00:16.045508 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:00:16.045508 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:00:16.045508 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:00:16.045508 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:00:16.046351 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:00:16.046351 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 05:00:16.046351 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:00:16.046351 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:00:16.046351 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:00:16.046351 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:00:16.046351 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:00:16.046351 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:00:16.046351 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:00:16.046351 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:00:16.046351 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 30 05:00:16.046351 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 05:00:16.046351 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 30 05:00:16.046351 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:00:16.050645 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 451/892] 10.222.0.77 () {56 vars in 1114 bytes} [Tue Aug 30 05:00:15 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 05:00:16.051628 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6763f973-67e6-4de2-aeb8-e798d67a61c6 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:16.065422 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:16.089577 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fca6c982-10be-4f86-9bfc-fbb42f25b31c None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:16.125955 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80bfb99c-b770-41fc-b04c-9b977b7c87cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:16.126702 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80bfb99c-b770-41fc-b04c-9b977b7c87cb None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:16.127108 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80bfb99c-b770-41fc-b04c-9b977b7c87cb None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:16.127562 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80bfb99c-b770-41fc-b04c-9b977b7c87cb None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:16.159535 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:16.168323 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6763f973-67e6-4de2-aeb8-e798d67a61c6 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:16.199243 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80bfb99c-b770-41fc-b04c-9b977b7c87cb None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:16.200824 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80bfb99c-b770-41fc-b04c-9b977b7c87cb None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:16.205374 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e36c19a9-a612-4f3c-891d-6ec34e724be3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:16.206160 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e36c19a9-a612-4f3c-891d-6ec34e724be3 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:16.206606 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e36c19a9-a612-4f3c-891d-6ec34e724be3 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:16.207039 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e36c19a9-a612-4f3c-891d-6ec34e724be3 None admin] PATH_INFO: `/v3/services` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:16.207799 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e36c19a9-a612-4f3c-891d-6ec34e724be3 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:16.208239 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e36c19a9-a612-4f3c-891d-6ec34e724be3 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:16.210394 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e36c19a9-a612-4f3c-891d-6ec34e724be3 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:16.220517 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 452/893] 10.222.0.77 () {56 vars in 1114 bytes} [Tue Aug 30 05:00:16 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:16.221904 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e36c19a9-a612-4f3c-891d-6ec34e724be3 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:16.283020 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:16.311242 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 442/894] 10.222.0.77 () {58 vars in 1282 bytes} [Tue Aug 30 05:00:15 2022] GET /identity/v3/auth/tokens => generated 1372 bytes in 331 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Aug 30 05:00:16.312664 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e2f6d85-5619-456c-ba56-a97b359bf816 None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:16.312947 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e2f6d85-5619-456c-ba56-a97b359bf816 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:16.313218 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e2f6d85-5619-456c-ba56-a97b359bf816 None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:16.314711 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 443/895] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:00:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:00:16.316819 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e2f6d85-5619-456c-ba56-a97b359bf816 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:16.337357 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e36c19a9-a612-4f3c-891d-6ec34e724be3 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:16.373828 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2704e3f2-c7cd-4290-b82b-a206a677cc21 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:16.374772 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2704e3f2-c7cd-4290-b82b-a206a677cc21 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:16.375328 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2704e3f2-c7cd-4290-b82b-a206a677cc21 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:16.375599 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2704e3f2-c7cd-4290-b82b-a206a677cc21 None admin] PATH_INFO: `/v3/services` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:16.376216 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2704e3f2-c7cd-4290-b82b-a206a677cc21 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:16.376512 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2704e3f2-c7cd-4290-b82b-a206a677cc21 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:16.378074 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2704e3f2-c7cd-4290-b82b-a206a677cc21 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:16.387786 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 453/896] 10.222.0.77 () {56 vars in 1114 bytes} [Tue Aug 30 05:00:16 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:16.387786 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff01e4a-5648-45be-940a-65bb7b029e14 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:16.387786 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff01e4a-5648-45be-940a-65bb7b029e14 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:16.387786 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff01e4a-5648-45be-940a-65bb7b029e14 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:16.403617 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-fff01e4a-5648-45be-940a-65bb7b029e14 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:00:16.404892 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-fff01e4a-5648-45be-940a-65bb7b029e14 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:00:16.423981 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:16.426603 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e2f6d85-5619-456c-ba56-a97b359bf816 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:16.469022 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44bc56dc-f3c0-417a-88d0-b201968d1e95 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:16.469641 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44bc56dc-f3c0-417a-88d0-b201968d1e95 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:16.469916 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44bc56dc-f3c0-417a-88d0-b201968d1e95 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:16.470179 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44bc56dc-f3c0-417a-88d0-b201968d1e95 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:16.482858 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 454/897] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:00:16 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 05:00:16.482858 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fff01e4a-5648-45be-940a-65bb7b029e14 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:16.483121 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44bc56dc-f3c0-417a-88d0-b201968d1e95 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:16.483121 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44bc56dc-f3c0-417a-88d0-b201968d1e95 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:16.509327 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-44bc56dc-f3c0-417a-88d0-b201968d1e95 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 05:00:16.509327 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:16.509327 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:00:16.509327 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:00:16.509327 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:00:16.509327 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:00:16.509327 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:00:16.509327 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:00:16.509327 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:00:16.509327 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:00:16.509327 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:00:16.509327 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:00:16.509327 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 05:00:16.509327 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 05:00:16.509327 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 05:00:16.509327 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 05:00:16.510501 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:00:16.510501 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:00:16.510501 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:00:16.510501 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:00:16.510501 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:00:16.510501 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:00:16.510501 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:00:16.510501 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 05:00:16.510501 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:00:16.510501 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:00:16.510501 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:00:16.510501 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:00:16.510501 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:00:16.510501 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:00:16.510501 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:00:16.510501 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:00:16.510501 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 05:00:16.511722 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 05:00:16.511722 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 05:00:16.511722 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:00:16.511722 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 444/898] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 05:00:16 2022] GET /identity/v3/domains/Default => generated 87 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:00:16.516597 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44bc56dc-f3c0-417a-88d0-b201968d1e95 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:16.549927 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:16.599013 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fff01e4a-5648-45be-940a-65bb7b029e14 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:16.627279 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44bc56dc-f3c0-417a-88d0-b201968d1e95 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:16.635736 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e97b0ad-4be5-4d1b-a7d7-3500e2fd1d32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:16.636385 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e97b0ad-4be5-4d1b-a7d7-3500e2fd1d32 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:16.636733 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e97b0ad-4be5-4d1b-a7d7-3500e2fd1d32 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:16.637070 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e97b0ad-4be5-4d1b-a7d7-3500e2fd1d32 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:16.637837 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e97b0ad-4be5-4d1b-a7d7-3500e2fd1d32 None admin] RBAC: Adding query filter params (interface=public, service_id=8a5bedbd0f724f9d888b1e8447243e63, region_id=RegionOne) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:16.638207 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e97b0ad-4be5-4d1b-a7d7-3500e2fd1d32 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:16.639664 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e97b0ad-4be5-4d1b-a7d7-3500e2fd1d32 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:16.645251 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 455/899] 10.222.0.77 () {56 vars in 1251 bytes} [Tue Aug 30 05:00:16 2022] GET /identity/v3/endpoints?service_id=8a5bedbd0f724f9d888b1e8447243e63&interface=public®ion_id=RegionOne => generated 194 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:16.650921 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e97b0ad-4be5-4d1b-a7d7-3500e2fd1d32 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:16.666238 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe082c4e-fa0c-4f8f-8cf2-a91d379974d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:16.667201 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe082c4e-fa0c-4f8f-8cf2-a91d379974d0 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:16.667493 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe082c4e-fa0c-4f8f-8cf2-a91d379974d0 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:16.667756 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe082c4e-fa0c-4f8f-8cf2-a91d379974d0 None admin] PATH_INFO: `/v3/domains` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:16.668391 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe082c4e-fa0c-4f8f-8cf2-a91d379974d0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:16.668697 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe082c4e-fa0c-4f8f-8cf2-a91d379974d0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:16.670051 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe082c4e-fa0c-4f8f-8cf2-a91d379974d0 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:16.672766 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:16.686482 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 445/900] 10.222.0.77 () {56 vars in 1109 bytes} [Tue Aug 30 05:00:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:16.687530 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe082c4e-fa0c-4f8f-8cf2-a91d379974d0 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:16.784611 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e97b0ad-4be5-4d1b-a7d7-3500e2fd1d32 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:16.794487 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:16.811334 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe082c4e-fa0c-4f8f-8cf2-a91d379974d0 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:16.821231 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e73e1095-7867-4433-89f6-5c64cec7f70d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:16.821836 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e73e1095-7867-4433-89f6-5c64cec7f70d None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:16.822116 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e73e1095-7867-4433-89f6-5c64cec7f70d None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:16.822855 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e73e1095-7867-4433-89f6-5c64cec7f70d None admin] PATH_INFO: `/v3/services` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:16.823472 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e73e1095-7867-4433-89f6-5c64cec7f70d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:16.825168 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e73e1095-7867-4433-89f6-5c64cec7f70d None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:16.831272 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 456/901] 10.222.0.77 () {56 vars in 1087 bytes} [Tue Aug 30 05:00:16 2022] GET /identity/v3/services => generated 1268 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:00:16.831840 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e73e1095-7867-4433-89f6-5c64cec7f70d None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:16.865365 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-737c52bf-3943-4c3b-9da2-58b65d2989ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:16.866745 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-737c52bf-3943-4c3b-9da2-58b65d2989ba None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:16.867252 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-737c52bf-3943-4c3b-9da2-58b65d2989ba None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:16.868178 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-737c52bf-3943-4c3b-9da2-58b65d2989ba None admin] PATH_INFO: `/v3/projects/service` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:16.878482 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-737c52bf-3943-4c3b-9da2-58b65d2989ba None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:16.881087 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-737c52bf-3943-4c3b-9da2-58b65d2989ba None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:16.899036 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-737c52bf-3943-4c3b-9da2-58b65d2989ba None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 05:00:16.899036 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:16.899036 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:00:16.899036 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:00:16.899036 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:00:16.899036 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:00:16.899036 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:00:16.899036 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:00:16.899036 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:00:16.899036 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:00:16.899036 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:00:16.899036 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:00:16.899036 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 05:00:16.899036 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 05:00:16.899036 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 05:00:16.899984 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 05:00:16.899984 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:00:16.899984 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:00:16.899984 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:00:16.899984 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:00:16.899984 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:00:16.899984 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:00:16.899984 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:00:16.899984 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 05:00:16.899984 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:00:16.899984 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:00:16.899984 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:00:16.899984 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:00:16.899984 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:00:16.899984 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:00:16.899984 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:00:16.899984 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:00:16.900861 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 05:00:16.900861 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 05:00:16.900861 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 05:00:16.900861 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 05:00:16.900861 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 05:00:16.900861 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 05:00:16.900861 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 05:00:16.900861 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:00:16.904000 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 446/902] 10.222.0.77 () {56 vars in 1111 bytes} [Tue Aug 30 05:00:16 2022] GET /identity/v3/projects/service => generated 88 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 05:00:16.915864 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-737c52bf-3943-4c3b-9da2-58b65d2989ba None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:16.936659 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:16.959425 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e73e1095-7867-4433-89f6-5c64cec7f70d None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:17.003580 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-641fe982-413f-442f-a866-c2e4fbb829e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:17.004182 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-641fe982-413f-442f-a866-c2e4fbb829e1 None admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:17.004463 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-641fe982-413f-442f-a866-c2e4fbb829e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:17.004733 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-641fe982-413f-442f-a866-c2e4fbb829e1 None admin] PATH_INFO: `/v3/services` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:17.005377 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-641fe982-413f-442f-a866-c2e4fbb829e1 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:17.007320 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-641fe982-413f-442f-a866-c2e4fbb829e1 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:17.023817 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 457/903] 10.222.0.77 () {60 vars in 1141 bytes} [Tue Aug 30 05:00:16 2022] POST /identity/v3/services => generated 241 bytes in 192 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:00:17.058647 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-737c52bf-3943-4c3b-9da2-58b65d2989ba None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:17.110270 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdb97f28-428c-44b5-889e-84a853869897 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:17.110802 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdb97f28-428c-44b5-889e-84a853869897 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:17.111105 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdb97f28-428c-44b5-889e-84a853869897 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:17.111377 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdb97f28-428c-44b5-889e-84a853869897 None admin] PATH_INFO: `/v3/projects` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:17.112137 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdb97f28-428c-44b5-889e-84a853869897 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:17.112443 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdb97f28-428c-44b5-889e-84a853869897 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:17.114000 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdb97f28-428c-44b5-889e-84a853869897 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:17.129263 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 447/904] 10.222.0.77 () {56 vars in 1148 bytes} [Tue Aug 30 05:00:16 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:17.135992 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-641fe982-413f-442f-a866-c2e4fbb829e1 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:17.146373 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-141cdbec-0c46-445e-9998-4ed15026ec56 None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:17.146669 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-141cdbec-0c46-445e-9998-4ed15026ec56 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:17.147941 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-141cdbec-0c46-445e-9998-4ed15026ec56 None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:17.148917 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 448/905] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:00:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:00:17.154056 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4eba464-d6b7-43d2-94b2-79632ec717e1 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:17.154338 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4eba464-d6b7-43d2-94b2-79632ec717e1 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:17.154613 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4eba464-d6b7-43d2-94b2-79632ec717e1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:17.171765 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:17.175882 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-b4eba464-d6b7-43d2-94b2-79632ec717e1 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:00:17.178160 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-b4eba464-d6b7-43d2-94b2-79632ec717e1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:00:17.263934 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-641fe982-413f-442f-a866-c2e4fbb829e1 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:17.270068 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 449/906] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:00:17 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 30 05:00:17.304489 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f9c076a-3c0b-40c2-9e00-9b45806ed183 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:17.305090 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f9c076a-3c0b-40c2-9e00-9b45806ed183 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:17.305369 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f9c076a-3c0b-40c2-9e00-9b45806ed183 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:17.305648 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f9c076a-3c0b-40c2-9e00-9b45806ed183 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:17.306357 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f9c076a-3c0b-40c2-9e00-9b45806ed183 None admin] RBAC: Adding query filter params (role.id=2d3c2095a28d4f4f8a8328c4b61b0015, scope.project.id=e9919141e3b947ab8d91b55bba4a7de6, user.id=fe51dbec91624ed1ab00b441e29a3dd5) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:17.306660 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f9c076a-3c0b-40c2-9e00-9b45806ed183 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:17.308116 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f9c076a-3c0b-40c2-9e00-9b45806ed183 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:17.314171 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 458/907] 10.222.0.77 () {56 vars in 1374 bytes} [Tue Aug 30 05:00:17 2022] GET /identity/v3/role_assignments?user.id=fe51dbec91624ed1ab00b441e29a3dd5&scope.project.id=e9919141e3b947ab8d91b55bba4a7de6&role.id=2d3c2095a28d4f4f8a8328c4b61b0015 => generated 259 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:17.544235 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4eba464-d6b7-43d2-94b2-79632ec717e1 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:17.646760 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4eba464-d6b7-43d2-94b2-79632ec717e1 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:17.683399 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4666b088-813c-4662-978c-add7f12055bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:17.684067 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4666b088-813c-4662-978c-add7f12055bc None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:17.684345 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4666b088-813c-4662-978c-add7f12055bc None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:17.684610 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4666b088-813c-4662-978c-add7f12055bc None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:17.685173 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4666b088-813c-4662-978c-add7f12055bc None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:17.686584 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4666b088-813c-4662-978c-add7f12055bc None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:17.688303 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:17.696954 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-4666b088-813c-4662-978c-add7f12055bc None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 30 05:00:17.696954 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:17.696954 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 30 05:00:17.696954 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 30 05:00:17.696954 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 30 05:00:17.696954 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 30 05:00:17.696954 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 30 05:00:17.696954 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 05:00:17.696954 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 30 05:00:17.696954 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:00:17.696954 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 05:00:17.696954 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:00:17.696954 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:17.696954 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:00:17.696954 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:00:17.696954 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:00:17.698070 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:00:17.698070 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:00:17.698070 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:00:17.698070 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:00:17.698070 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:00:17.698070 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:00:17.698070 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:00:17.698070 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 30 05:00:17.698070 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 30 05:00:17.698070 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 30 05:00:17.698070 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 30 05:00:17.698070 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:00:17.698070 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:00:17.698070 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:00:17.698070 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:00:17.698070 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:00:17.698070 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:00:17.699200 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:00:17.699200 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 05:00:17.699200 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:00:17.699200 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:00:17.699200 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:00:17.699200 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:00:17.699200 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:00:17.699200 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:00:17.699200 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:00:17.699200 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:00:17.699200 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 30 05:00:17.699200 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 05:00:17.699200 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 30 05:00:17.699200 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:00:17.699200 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 450/908] 10.222.0.77 () {56 vars in 1132 bytes} [Tue Aug 30 05:00:17 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:00:17.704093 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f9c076a-3c0b-40c2-9e00-9b45806ed183 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:17.713880 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:17.813992 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f9c076a-3c0b-40c2-9e00-9b45806ed183 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:17.849644 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c84677fe-79d2-430a-9d2a-748ee5c98389 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:17.850994 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c84677fe-79d2-430a-9d2a-748ee5c98389 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:17.851380 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c84677fe-79d2-430a-9d2a-748ee5c98389 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:17.851754 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c84677fe-79d2-430a-9d2a-748ee5c98389 None admin] PATH_INFO: `/v3/services` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:17.852478 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c84677fe-79d2-430a-9d2a-748ee5c98389 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:17.853140 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c84677fe-79d2-430a-9d2a-748ee5c98389 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:17.854610 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c84677fe-79d2-430a-9d2a-748ee5c98389 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:17.860072 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 459/909] 10.222.0.77 () {56 vars in 1126 bytes} [Tue Aug 30 05:00:17 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:17.865405 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4666b088-813c-4662-978c-add7f12055bc None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:17.968194 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4666b088-813c-4662-978c-add7f12055bc None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:18.004400 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef7da7df-b294-49eb-9dc1-40ebe3ef22b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:18.005019 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef7da7df-b294-49eb-9dc1-40ebe3ef22b2 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:18.005298 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef7da7df-b294-49eb-9dc1-40ebe3ef22b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:18.005564 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef7da7df-b294-49eb-9dc1-40ebe3ef22b2 None admin] PATH_INFO: `/v3/services` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:18.006207 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef7da7df-b294-49eb-9dc1-40ebe3ef22b2 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:18.006551 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef7da7df-b294-49eb-9dc1-40ebe3ef22b2 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:18.007958 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef7da7df-b294-49eb-9dc1-40ebe3ef22b2 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:18.013784 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 451/910] 10.222.0.77 () {56 vars in 1126 bytes} [Tue Aug 30 05:00:17 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:18.019393 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c84677fe-79d2-430a-9d2a-748ee5c98389 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:18.091260 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:18.125173 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c84677fe-79d2-430a-9d2a-748ee5c98389 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:18.160925 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2be0842-9243-447e-b2a9-8749796479e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:18.161515 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2be0842-9243-447e-b2a9-8749796479e2 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:18.161790 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2be0842-9243-447e-b2a9-8749796479e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:18.162059 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2be0842-9243-447e-b2a9-8749796479e2 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:18.162777 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2be0842-9243-447e-b2a9-8749796479e2 None admin] RBAC: Adding query filter params (interface=public, service_id=7be814dc001a4ea6ab750486dbd4000b, region_id=RegionOne) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:18.163111 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2be0842-9243-447e-b2a9-8749796479e2 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:18.164477 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2be0842-9243-447e-b2a9-8749796479e2 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:18.169958 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 460/911] 10.222.0.77 () {56 vars in 1251 bytes} [Tue Aug 30 05:00:18 2022] GET /identity/v3/endpoints?service_id=7be814dc001a4ea6ab750486dbd4000b&interface=public®ion_id=RegionOne => generated 194 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:18.176171 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef7da7df-b294-49eb-9dc1-40ebe3ef22b2 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:18.259177 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:18.280779 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef7da7df-b294-49eb-9dc1-40ebe3ef22b2 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:18.318650 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-383d4330-914f-47df-b27b-844b52b2cf92 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:18.320173 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-383d4330-914f-47df-b27b-844b52b2cf92 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:18.320453 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-383d4330-914f-47df-b27b-844b52b2cf92 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:18.320726 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-383d4330-914f-47df-b27b-844b52b2cf92 None admin] PATH_INFO: `/v3/services` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:18.321322 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-383d4330-914f-47df-b27b-844b52b2cf92 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:18.322683 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-383d4330-914f-47df-b27b-844b52b2cf92 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:18.329129 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 452/912] 10.222.0.77 () {56 vars in 1087 bytes} [Tue Aug 30 05:00:18 2022] GET /identity/v3/services => generated 1497 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:00:18.877607 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-011b7358-4ed4-4736-921c-f0abd052191d None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:18.878136 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-011b7358-4ed4-4736-921c-f0abd052191d None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:18.878439 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-011b7358-4ed4-4736-921c-f0abd052191d None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:18.880188 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 461/913] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:00:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:00:18.885930 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6242d7c0-210c-4bb2-afc8-0fdd13c18257 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:18.886338 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6242d7c0-210c-4bb2-afc8-0fdd13c18257 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:18.886669 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6242d7c0-210c-4bb2-afc8-0fdd13c18257 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:18.905776 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-6242d7c0-210c-4bb2-afc8-0fdd13c18257 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:00:18.907005 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-6242d7c0-210c-4bb2-afc8-0fdd13c18257 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:00:18.982994 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 453/914] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:00:18 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 05:00:19.072133 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebe57e77-ce18-4a46-a183-403d9dae1786 None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:19.072441 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebe57e77-ce18-4a46-a183-403d9dae1786 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:19.072686 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebe57e77-ce18-4a46-a183-403d9dae1786 None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:19.074316 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 462/915] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:00:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:00:19.079649 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed3d7d90-0e00-4743-b996-499eea9dcdbf None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:19.079938 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed3d7d90-0e00-4743-b996-499eea9dcdbf None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:19.080214 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed3d7d90-0e00-4743-b996-499eea9dcdbf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:19.098933 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-ed3d7d90-0e00-4743-b996-499eea9dcdbf None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:00:19.100122 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-ed3d7d90-0e00-4743-b996-499eea9dcdbf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:00:19.152107 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:19.174929 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 454/916] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:00:19 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 05:00:19.252046 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebe57e77-ce18-4a46-a183-403d9dae1786 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:19.333471 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2da7225c-d48c-4a3e-947c-aef614b306ad None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:19.333757 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2da7225c-d48c-4a3e-947c-aef614b306ad None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:19.334030 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2da7225c-d48c-4a3e-947c-aef614b306ad None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:19.335618 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 455/917] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:00:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 05:00:19.341427 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b11af6e9-9e97-4e07-978e-10adf927ba94 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:19.341717 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b11af6e9-9e97-4e07-978e-10adf927ba94 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:19.341994 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b11af6e9-9e97-4e07-978e-10adf927ba94 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:19.360239 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-b11af6e9-9e97-4e07-978e-10adf927ba94 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:00:19.361443 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-b11af6e9-9e97-4e07-978e-10adf927ba94 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:00:19.365269 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebe57e77-ce18-4a46-a183-403d9dae1786 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:19.407548 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9f8da32-29ca-437d-8f6b-f1651d373663 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:19.408459 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9f8da32-29ca-437d-8f6b-f1651d373663 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:19.409278 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9f8da32-29ca-437d-8f6b-f1651d373663 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:19.409753 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9f8da32-29ca-437d-8f6b-f1651d373663 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:19.410837 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9f8da32-29ca-437d-8f6b-f1651d373663 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:19.412749 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9f8da32-29ca-437d-8f6b-f1651d373663 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:19.423971 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-b9f8da32-29ca-437d-8f6b-f1651d373663 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 30 05:00:19.423971 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:19.423971 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 30 05:00:19.423971 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 30 05:00:19.423971 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 30 05:00:19.423971 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 30 05:00:19.423971 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 30 05:00:19.423971 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 05:00:19.423971 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 30 05:00:19.423971 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:00:19.423971 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 05:00:19.423971 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:00:19.423971 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:19.423971 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:00:19.423971 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:00:19.423971 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:00:19.423971 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:00:19.424992 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:00:19.424992 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:00:19.424992 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:00:19.424992 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:00:19.424992 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:00:19.424992 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:00:19.424992 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 30 05:00:19.424992 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 30 05:00:19.424992 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 30 05:00:19.424992 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 30 05:00:19.424992 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:00:19.424992 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:00:19.424992 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:00:19.424992 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:00:19.424992 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:00:19.424992 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:00:19.425830 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:00:19.425830 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 05:00:19.425830 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:00:19.425830 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:00:19.425830 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:00:19.425830 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:00:19.425830 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:00:19.425830 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:00:19.425830 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:00:19.425830 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:00:19.425830 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 30 05:00:19.425830 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 05:00:19.425830 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 30 05:00:19.425830 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:00:19.429188 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 463/918] 10.222.0.77 () {56 vars in 1114 bytes} [Tue Aug 30 05:00:19 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:00:19.431008 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:19.437020 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9f8da32-29ca-437d-8f6b-f1651d373663 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:19.444193 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 456/919] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:00:19 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 30 05:00:19.450130 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b11af6e9-9e97-4e07-978e-10adf927ba94 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:19.566759 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b11af6e9-9e97-4e07-978e-10adf927ba94 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:19.577213 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9f8da32-29ca-437d-8f6b-f1651d373663 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:19.588056 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:19.608414 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8c826bb-da35-4809-b533-deb28406f5ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:19.609216 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c826bb-da35-4809-b533-deb28406f5ed None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:19.609592 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c826bb-da35-4809-b533-deb28406f5ed None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:19.610056 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c826bb-da35-4809-b533-deb28406f5ed None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:19.610727 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8c826bb-da35-4809-b533-deb28406f5ed None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:19.612212 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8c826bb-da35-4809-b533-deb28406f5ed None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:19.621853 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6661a4b4-9a18-4e1f-a9a4-7a1f0e69c14f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:19.622449 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6661a4b4-9a18-4e1f-a9a4-7a1f0e69c14f None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:19.622725 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6661a4b4-9a18-4e1f-a9a4-7a1f0e69c14f None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:19.623023 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6661a4b4-9a18-4e1f-a9a4-7a1f0e69c14f None admin] PATH_INFO: `/v3/services` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:19.623643 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6661a4b4-9a18-4e1f-a9a4-7a1f0e69c14f None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:19.623940 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6661a4b4-9a18-4e1f-a9a4-7a1f0e69c14f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:19.624355 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-a8c826bb-da35-4809-b533-deb28406f5ed None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 30 05:00:19.624355 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:19.624355 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 30 05:00:19.624355 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 30 05:00:19.624355 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 30 05:00:19.624355 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 30 05:00:19.624355 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 30 05:00:19.624355 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 05:00:19.624355 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 30 05:00:19.624355 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:00:19.624355 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 05:00:19.624355 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:00:19.624355 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:19.624355 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:00:19.624355 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:00:19.624355 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:00:19.624355 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:00:19.625501 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:00:19.625501 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:00:19.625501 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:00:19.625501 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:00:19.625501 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:00:19.625501 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:00:19.625501 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 30 05:00:19.625501 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 30 05:00:19.625501 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 30 05:00:19.625501 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 30 05:00:19.625501 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:00:19.625501 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:00:19.625501 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:00:19.625501 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:00:19.625501 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:00:19.625501 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:00:19.626525 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:00:19.626525 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 05:00:19.626525 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:00:19.626525 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:00:19.626525 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:00:19.626525 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:00:19.626525 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:00:19.626525 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:00:19.626525 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:00:19.626525 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:00:19.626525 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 30 05:00:19.626525 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 05:00:19.626525 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 30 05:00:19.626525 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:00:19.627445 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6661a4b4-9a18-4e1f-a9a4-7a1f0e69c14f None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:19.627592 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 457/920] 10.222.0.77 () {56 vars in 1126 bytes} [Tue Aug 30 05:00:19 2022] GET /identity/v3/services/object-store => generated 93 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 05:00:19.633799 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8c826bb-da35-4809-b533-deb28406f5ed None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:19.636611 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 464/921] 10.222.0.77 () {56 vars in 1114 bytes} [Tue Aug 30 05:00:19 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 05:00:19.644422 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6661a4b4-9a18-4e1f-a9a4-7a1f0e69c14f None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:19.711477 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:19.754707 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6661a4b4-9a18-4e1f-a9a4-7a1f0e69c14f None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:19.764402 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8c826bb-da35-4809-b533-deb28406f5ed None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:19.788719 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75603321-12df-4e7d-824a-3de6b92d1af1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:19.789321 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75603321-12df-4e7d-824a-3de6b92d1af1 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:19.789595 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75603321-12df-4e7d-824a-3de6b92d1af1 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:19.789859 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75603321-12df-4e7d-824a-3de6b92d1af1 None admin] PATH_INFO: `/v3/services` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:19.790476 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75603321-12df-4e7d-824a-3de6b92d1af1 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:19.790781 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75603321-12df-4e7d-824a-3de6b92d1af1 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:19.792152 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75603321-12df-4e7d-824a-3de6b92d1af1 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:19.796868 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 465/922] 10.222.0.77 () {56 vars in 1114 bytes} [Tue Aug 30 05:00:19 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:19.797633 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75603321-12df-4e7d-824a-3de6b92d1af1 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:19.806521 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbb87f18-852f-4716-a509-acb2f847d064 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:19.807421 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb87f18-852f-4716-a509-acb2f847d064 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:19.807987 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb87f18-852f-4716-a509-acb2f847d064 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:19.808460 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb87f18-852f-4716-a509-acb2f847d064 None admin] PATH_INFO: `/v3/services` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:19.809264 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbb87f18-852f-4716-a509-acb2f847d064 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:19.809707 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbb87f18-852f-4716-a509-acb2f847d064 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:19.811476 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbb87f18-852f-4716-a509-acb2f847d064 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:19.821522 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 458/923] 10.222.0.77 () {56 vars in 1122 bytes} [Tue Aug 30 05:00:19 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:19.822352 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbb87f18-852f-4716-a509-acb2f847d064 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:19.871847 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:19.904636 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75603321-12df-4e7d-824a-3de6b92d1af1 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:19.931565 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbb87f18-852f-4716-a509-acb2f847d064 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:19.932479 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:19.946778 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00414a1a-6457-49cf-b968-97d835a83721 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:19.947432 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00414a1a-6457-49cf-b968-97d835a83721 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:19.947709 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00414a1a-6457-49cf-b968-97d835a83721 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:19.947978 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00414a1a-6457-49cf-b968-97d835a83721 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:19.958231 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00414a1a-6457-49cf-b968-97d835a83721 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 05:00:19.963618 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-00414a1a-6457-49cf-b968-97d835a83721 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 30 05:00:19.963618 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:19.963618 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:00:19.963618 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:00:19.963618 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:00:19.963618 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:00:19.963618 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:00:19.963618 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:00:19.963618 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:00:19.963618 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:00:19.963618 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:00:19.963618 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:00:19.963618 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 05:00:19.963618 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 05:00:19.963618 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 05:00:19.963618 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 05:00:19.964723 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 05:00:19.964723 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 05:00:19.964723 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 05:00:19.964723 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 05:00:19.964723 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:00:19.964723 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:00:19.964723 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:00:19.964723 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:00:19.964723 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:00:19.964723 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:00:19.964723 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:00:19.964723 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 05:00:19.964723 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:00:19.964723 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:00:19.964723 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:00:19.964723 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:00:19.965567 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:00:19.965567 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:00:19.965567 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:00:19.965567 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:00:19.965567 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 05:00:19.965567 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 05:00:19.965567 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 05:00:19.965567 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 05:00:19.965567 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 05:00:19.965567 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 05:00:19.965567 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 30 05:00:19.965567 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:00:19.965567 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 466/924] 10.222.0.77 () {56 vars in 1120 bytes} [Tue Aug 30 05:00:19 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:00:19.965567 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00414a1a-6457-49cf-b968-97d835a83721 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:19.978362 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd6c8ece-c2c7-4c2b-b43f-16444438a270 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:19.979274 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd6c8ece-c2c7-4c2b-b43f-16444438a270 None admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:19.979813 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd6c8ece-c2c7-4c2b-b43f-16444438a270 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:19.980245 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd6c8ece-c2c7-4c2b-b43f-16444438a270 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:19.981116 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd6c8ece-c2c7-4c2b-b43f-16444438a270 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:19.982937 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd6c8ece-c2c7-4c2b-b43f-16444438a270 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:20.004551 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 459/925] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:00:19 2022] POST /identity/v3/endpoints => generated 346 bytes in 182 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:00:20.005362 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd6c8ece-c2c7-4c2b-b43f-16444438a270 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:20.109332 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00414a1a-6457-49cf-b968-97d835a83721 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:20.128345 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:20.139227 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd6c8ece-c2c7-4c2b-b43f-16444438a270 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:20.164736 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bce82456-32c1-4d0d-9cd1-84ee80243560 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:20.166515 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bce82456-32c1-4d0d-9cd1-84ee80243560 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:20.167003 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bce82456-32c1-4d0d-9cd1-84ee80243560 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:20.168347 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bce82456-32c1-4d0d-9cd1-84ee80243560 None admin] PATH_INFO: `/v3/services` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:20.171020 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bce82456-32c1-4d0d-9cd1-84ee80243560 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:20.171627 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bce82456-32c1-4d0d-9cd1-84ee80243560 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:20.173020 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bce82456-32c1-4d0d-9cd1-84ee80243560 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:20.178745 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 467/926] 10.222.0.77 () {56 vars in 1122 bytes} [Tue Aug 30 05:00:19 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:20.183754 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-242ac897-1088-45dc-bd7e-1aceb6d249e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:20.184382 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bce82456-32c1-4d0d-9cd1-84ee80243560 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:20.184709 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-242ac897-1088-45dc-bd7e-1aceb6d249e1 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:20.185143 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-242ac897-1088-45dc-bd7e-1aceb6d249e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:20.185413 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-242ac897-1088-45dc-bd7e-1aceb6d249e1 None admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:20.186119 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-242ac897-1088-45dc-bd7e-1aceb6d249e1 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:20.186426 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-242ac897-1088-45dc-bd7e-1aceb6d249e1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:20.187898 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-242ac897-1088-45dc-bd7e-1aceb6d249e1 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:20.198017 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 460/927] 10.222.0.77 () {56 vars in 1115 bytes} [Tue Aug 30 05:00:20 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:20.208114 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-242ac897-1088-45dc-bd7e-1aceb6d249e1 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:20.254438 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:20.333229 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bce82456-32c1-4d0d-9cd1-84ee80243560 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:20.334382 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-242ac897-1088-45dc-bd7e-1aceb6d249e1 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:20.369763 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e315198-5da5-4d74-ab29-c39c4427d22d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:20.370466 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e315198-5da5-4d74-ab29-c39c4427d22d None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:20.370858 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e315198-5da5-4d74-ab29-c39c4427d22d None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:20.371248 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e315198-5da5-4d74-ab29-c39c4427d22d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:20.372331 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-172b9fbc-e868-4b21-a504-160683ed50e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:20.373001 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-172b9fbc-e868-4b21-a504-160683ed50e9 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:20.373353 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-172b9fbc-e868-4b21-a504-160683ed50e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:20.373696 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-172b9fbc-e868-4b21-a504-160683ed50e9 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:20.374465 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-172b9fbc-e868-4b21-a504-160683ed50e9 None admin] RBAC: Adding query filter params (interface=public, service_id=f46ae468494244dfac7071ab1f2efd69, region_id=RegionOne) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:20.374865 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-172b9fbc-e868-4b21-a504-160683ed50e9 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:20.376293 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-172b9fbc-e868-4b21-a504-160683ed50e9 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:20.382897 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 468/928] 10.222.0.77 () {56 vars in 1251 bytes} [Tue Aug 30 05:00:20 2022] GET /identity/v3/endpoints?service_id=f46ae468494244dfac7071ab1f2efd69&interface=public®ion_id=RegionOne => generated 194 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 05:00:20.383050 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e315198-5da5-4d74-ab29-c39c4427d22d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:20.384079 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d2b3d03-62d8-436e-b4d2-01d3e4e7eff7 None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:20.384436 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d2b3d03-62d8-436e-b4d2-01d3e4e7eff7 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:20.384776 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e315198-5da5-4d74-ab29-c39c4427d22d None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:20.384894 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d2b3d03-62d8-436e-b4d2-01d3e4e7eff7 None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:20.386304 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 469/929] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:00:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:00:20.390843 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d2b3d03-62d8-436e-b4d2-01d3e4e7eff7 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:20.400627 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-2e315198-5da5-4d74-ab29-c39c4427d22d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 05:00:20.400627 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:20.400627 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:00:20.400627 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:00:20.400627 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:00:20.400627 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:00:20.400627 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:00:20.400627 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:00:20.400627 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:00:20.400627 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:00:20.400627 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:00:20.400627 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:00:20.400627 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 05:00:20.400627 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 05:00:20.400627 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 05:00:20.400627 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 05:00:20.401687 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:00:20.401687 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:00:20.401687 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:00:20.401687 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:00:20.401687 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:00:20.401687 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:00:20.401687 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:00:20.401687 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 05:00:20.401687 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:00:20.401687 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:00:20.401687 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:00:20.401687 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:00:20.401687 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:00:20.401687 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:00:20.401687 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:00:20.401687 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:00:20.401687 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 05:00:20.402689 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 05:00:20.402689 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 05:00:20.402689 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:00:20.402689 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 461/930] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 05:00:20 2022] GET /identity/v3/domains/Default => generated 87 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 05:00:20.403072 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f516a8b1-e0bb-4e9b-862f-9cb87ba16c6b None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:20.403709 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f516a8b1-e0bb-4e9b-862f-9cb87ba16c6b None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:20.404080 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f516a8b1-e0bb-4e9b-862f-9cb87ba16c6b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:20.422552 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-f516a8b1-e0bb-4e9b-862f-9cb87ba16c6b None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:00:20.423920 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-f516a8b1-e0bb-4e9b-862f-9cb87ba16c6b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:00:20.428041 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:20.504384 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 462/931] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:00:20 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 05:00:20.505566 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f516a8b1-e0bb-4e9b-862f-9cb87ba16c6b None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:20.524012 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d2b3d03-62d8-436e-b4d2-01d3e4e7eff7 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:20.545345 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:20.568783 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c7486f3-789c-4e48-8aef-352da5d69646 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:20.570073 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c7486f3-789c-4e48-8aef-352da5d69646 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:20.570726 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c7486f3-789c-4e48-8aef-352da5d69646 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:20.570726 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c7486f3-789c-4e48-8aef-352da5d69646 None admin] PATH_INFO: `/v3/services` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:20.571418 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c7486f3-789c-4e48-8aef-352da5d69646 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:20.572847 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c7486f3-789c-4e48-8aef-352da5d69646 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:20.579731 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 470/932] 10.222.0.77 () {56 vars in 1087 bytes} [Tue Aug 30 05:00:20 2022] GET /identity/v3/services => generated 1497 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:00:20.633627 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f516a8b1-e0bb-4e9b-862f-9cb87ba16c6b None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:20.673662 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be8b5222-8069-4eac-85b4-9e5ebf292f0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:20.674438 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be8b5222-8069-4eac-85b4-9e5ebf292f0a None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:20.674848 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be8b5222-8069-4eac-85b4-9e5ebf292f0a None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:20.675273 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be8b5222-8069-4eac-85b4-9e5ebf292f0a None admin] PATH_INFO: `/v3/domains` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:20.676097 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be8b5222-8069-4eac-85b4-9e5ebf292f0a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:20.676551 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be8b5222-8069-4eac-85b4-9e5ebf292f0a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:20.678227 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be8b5222-8069-4eac-85b4-9e5ebf292f0a None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:20.688881 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:20.693861 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 463/933] 10.222.0.77 () {56 vars in 1109 bytes} [Tue Aug 30 05:00:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:20.700939 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c7486f3-789c-4e48-8aef-352da5d69646 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:20.782360 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be8b5222-8069-4eac-85b4-9e5ebf292f0a None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:20.813007 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c7486f3-789c-4e48-8aef-352da5d69646 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:20.851321 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d608dce8-36e4-428e-a409-7685be842967 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:20.852221 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d608dce8-36e4-428e-a409-7685be842967 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:20.852671 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d608dce8-36e4-428e-a409-7685be842967 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:20.853197 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d608dce8-36e4-428e-a409-7685be842967 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:20.860600 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d608dce8-36e4-428e-a409-7685be842967 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:20.862053 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d608dce8-36e4-428e-a409-7685be842967 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:20.874579 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-d608dce8-36e4-428e-a409-7685be842967 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 30 05:00:20.874579 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:20.874579 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:00:20.874579 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:00:20.874579 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:00:20.874579 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:00:20.874579 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:00:20.874579 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:00:20.874579 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:00:20.874579 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:00:20.874579 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:00:20.874579 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:00:20.874579 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 05:00:20.874579 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 05:00:20.874579 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 05:00:20.874579 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 05:00:20.875864 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:00:20.875864 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:00:20.875864 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 05:00:20.875864 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 05:00:20.875864 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 05:00:20.875864 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 05:00:20.875864 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:00:20.875864 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:00:20.875864 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:00:20.875864 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:00:20.875864 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:00:20.875864 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 05:00:20.875864 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:00:20.875864 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:00:20.875864 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:00:20.875864 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:00:20.875864 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:00:20.875864 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:00:20.877170 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:00:20.877170 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:00:20.877170 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 05:00:20.877170 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 05:00:20.877170 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 05:00:20.877170 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 05:00:20.877170 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 05:00:20.877170 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 05:00:20.877170 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 30 05:00:20.877170 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:00:20.880651 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 471/934] 10.222.0.77 () {56 vars in 1117 bytes} [Tue Aug 30 05:00:20 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:00:20.883093 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d608dce8-36e4-428e-a409-7685be842967 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:20.915715 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be8b5222-8069-4eac-85b4-9e5ebf292f0a None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:20.925384 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:20.963790 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76c74129-5bf6-4102-bfc8-94b8012d5365 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:20.965519 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c74129-5bf6-4102-bfc8-94b8012d5365 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:20.965959 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c74129-5bf6-4102-bfc8-94b8012d5365 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:20.966380 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c74129-5bf6-4102-bfc8-94b8012d5365 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:20.967571 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76c74129-5bf6-4102-bfc8-94b8012d5365 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:20.969772 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76c74129-5bf6-4102-bfc8-94b8012d5365 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:20.982140 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-76c74129-5bf6-4102-bfc8-94b8012d5365 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 30 05:00:20.982140 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:20.982140 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 30 05:00:20.982140 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 30 05:00:20.982140 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 30 05:00:20.982140 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 30 05:00:20.982140 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 30 05:00:20.982140 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 05:00:20.982140 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 30 05:00:20.982140 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:00:20.982140 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 05:00:20.982140 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:00:20.982140 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:20.982140 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:00:20.982140 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:00:20.982140 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:00:20.983362 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:00:20.983362 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:00:20.983362 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:00:20.983362 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:00:20.983362 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:00:20.983362 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:00:20.983362 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:00:20.983362 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 30 05:00:20.983362 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 30 05:00:20.983362 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 30 05:00:20.983362 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 30 05:00:20.983362 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:00:20.983362 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:00:20.983362 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:00:20.983362 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:00:20.983362 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:00:20.983362 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:00:20.984446 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:00:20.984446 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 05:00:20.984446 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:00:20.984446 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:00:20.984446 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:00:20.984446 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:00:20.984446 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:00:20.984446 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:00:20.984446 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:00:20.984446 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:00:20.984446 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 30 05:00:20.984446 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 05:00:20.984446 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 30 05:00:20.984446 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:00:20.992182 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 464/935] 10.222.0.77 () {56 vars in 1132 bytes} [Tue Aug 30 05:00:20 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 05:00:20.994843 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76c74129-5bf6-4102-bfc8-94b8012d5365 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:21.002166 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d608dce8-36e4-428e-a409-7685be842967 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:21.042256 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b08efc3-d676-47e7-8506-97c95a1dee2d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:21.042783 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b08efc3-d676-47e7-8506-97c95a1dee2d None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:21.043104 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b08efc3-d676-47e7-8506-97c95a1dee2d None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:21.043377 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b08efc3-d676-47e7-8506-97c95a1dee2d None admin] PATH_INFO: `/v3/users` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:21.044106 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b08efc3-d676-47e7-8506-97c95a1dee2d None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:21.044508 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b08efc3-d676-47e7-8506-97c95a1dee2d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:21.045950 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b08efc3-d676-47e7-8506-97c95a1dee2d None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:21.065866 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 472/936] 10.222.0.77 () {56 vars in 1149 bytes} [Tue Aug 30 05:00:20 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:21.072247 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:21.072677 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b08efc3-d676-47e7-8506-97c95a1dee2d None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:21.125059 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76c74129-5bf6-4102-bfc8-94b8012d5365 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:21.177384 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3bc138c-d45c-4bc3-84ae-457505c28d93 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:21.177993 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3bc138c-d45c-4bc3-84ae-457505c28d93 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:21.178385 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3bc138c-d45c-4bc3-84ae-457505c28d93 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:21.181213 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3bc138c-d45c-4bc3-84ae-457505c28d93 None admin] PATH_INFO: `/v3/services` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:21.181868 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3bc138c-d45c-4bc3-84ae-457505c28d93 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:21.182551 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3bc138c-d45c-4bc3-84ae-457505c28d93 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:21.183965 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3bc138c-d45c-4bc3-84ae-457505c28d93 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:21.194855 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 465/937] 10.222.0.77 () {56 vars in 1126 bytes} [Tue Aug 30 05:00:20 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:21.196664 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3bc138c-d45c-4bc3-84ae-457505c28d93 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:21.197723 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:21.205612 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b08efc3-d676-47e7-8506-97c95a1dee2d None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:21.243555 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de10c2ed-09eb-4417-b9b4-93cb578f46b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:21.244368 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de10c2ed-09eb-4417-b9b4-93cb578f46b6 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:21.244761 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de10c2ed-09eb-4417-b9b4-93cb578f46b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:21.245098 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de10c2ed-09eb-4417-b9b4-93cb578f46b6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:21.256832 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de10c2ed-09eb-4417-b9b4-93cb578f46b6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:21.258531 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de10c2ed-09eb-4417-b9b4-93cb578f46b6 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:21.280250 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-de10c2ed-09eb-4417-b9b4-93cb578f46b6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 05:00:21.280250 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:21.280250 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:00:21.280250 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:00:21.280250 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:00:21.280250 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:00:21.280250 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:00:21.280250 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:00:21.280250 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:00:21.280250 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:00:21.280250 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:00:21.280250 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:00:21.280250 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 05:00:21.280250 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 05:00:21.280250 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 05:00:21.280250 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 05:00:21.281471 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:00:21.281471 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:00:21.281471 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:00:21.281471 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:00:21.281471 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:00:21.281471 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:00:21.281471 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:00:21.281471 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 05:00:21.281471 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:00:21.281471 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:00:21.281471 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:00:21.281471 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:00:21.281471 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:00:21.281471 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:00:21.281471 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:00:21.281471 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:00:21.281471 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 05:00:21.282569 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 05:00:21.282569 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 05:00:21.282569 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:00:21.288946 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 473/938] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 05:00:21 2022] GET /identity/v3/domains/Default => generated 87 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:00:21.289878 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de10c2ed-09eb-4417-b9b4-93cb578f46b6 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:21.331130 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3bc138c-d45c-4bc3-84ae-457505c28d93 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:21.360567 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:21.373052 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e83eb1f-02ed-4ab5-9b03-23c70ad09035 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:21.373659 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e83eb1f-02ed-4ab5-9b03-23c70ad09035 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:21.373938 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e83eb1f-02ed-4ab5-9b03-23c70ad09035 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:21.374209 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e83eb1f-02ed-4ab5-9b03-23c70ad09035 None admin] PATH_INFO: `/v3/services` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:21.374857 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e83eb1f-02ed-4ab5-9b03-23c70ad09035 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:21.375157 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e83eb1f-02ed-4ab5-9b03-23c70ad09035 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:21.376776 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e83eb1f-02ed-4ab5-9b03-23c70ad09035 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:21.388204 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 466/939] 10.222.0.77 () {56 vars in 1126 bytes} [Tue Aug 30 05:00:21 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 05:00:21.389224 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e83eb1f-02ed-4ab5-9b03-23c70ad09035 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:21.426077 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de10c2ed-09eb-4417-b9b4-93cb578f46b6 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:21.469882 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54f03308-a1cc-4fba-9040-3e85b16cef9b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:21.470487 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f03308-a1cc-4fba-9040-3e85b16cef9b None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:21.470764 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f03308-a1cc-4fba-9040-3e85b16cef9b None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:21.471059 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f03308-a1cc-4fba-9040-3e85b16cef9b None admin] PATH_INFO: `/v3/domains` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:21.471692 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54f03308-a1cc-4fba-9040-3e85b16cef9b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:21.472012 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54f03308-a1cc-4fba-9040-3e85b16cef9b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:21.474047 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54f03308-a1cc-4fba-9040-3e85b16cef9b None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:21.489124 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 474/940] 10.222.0.77 () {56 vars in 1109 bytes} [Tue Aug 30 05:00:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:21.495716 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54f03308-a1cc-4fba-9040-3e85b16cef9b None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:21.506583 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e83eb1f-02ed-4ab5-9b03-23c70ad09035 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:21.508637 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:21.552002 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71504100-71e6-4663-b79a-08c2ebaff157 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:21.553347 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71504100-71e6-4663-b79a-08c2ebaff157 None admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:21.553677 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71504100-71e6-4663-b79a-08c2ebaff157 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:21.554224 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71504100-71e6-4663-b79a-08c2ebaff157 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:21.554991 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71504100-71e6-4663-b79a-08c2ebaff157 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:21.557030 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71504100-71e6-4663-b79a-08c2ebaff157 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:21.588226 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 467/941] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:00:21 2022] POST /identity/v3/endpoints => generated 347 bytes in 199 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:00:21.617275 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54f03308-a1cc-4fba-9040-3e85b16cef9b None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:21.664804 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcf95020-1af8-46d6-9ecc-34140ed5942e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:21.665457 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf95020-1af8-46d6-9ecc-34140ed5942e None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:21.665736 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf95020-1af8-46d6-9ecc-34140ed5942e None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:21.666071 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf95020-1af8-46d6-9ecc-34140ed5942e None admin] PATH_INFO: `/v3/projects/service` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:21.675268 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcf95020-1af8-46d6-9ecc-34140ed5942e None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:21.676782 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcf95020-1af8-46d6-9ecc-34140ed5942e None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:21.690727 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-bcf95020-1af8-46d6-9ecc-34140ed5942e None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 05:00:21.690727 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:21.690727 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:00:21.690727 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:00:21.690727 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:00:21.690727 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:00:21.690727 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:00:21.690727 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:00:21.690727 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:00:21.690727 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:00:21.690727 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:00:21.690727 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:00:21.690727 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 05:00:21.690727 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 05:00:21.690727 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 05:00:21.691728 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 05:00:21.691728 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:00:21.691728 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:00:21.691728 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:00:21.691728 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:00:21.691728 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:00:21.691728 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:00:21.691728 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:00:21.691728 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 05:00:21.691728 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:00:21.691728 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:00:21.691728 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:00:21.691728 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:00:21.691728 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:00:21.691728 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:00:21.691728 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:00:21.691728 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:00:21.692605 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 05:00:21.692605 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 05:00:21.692605 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 05:00:21.692605 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 05:00:21.692605 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 05:00:21.692605 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 05:00:21.692605 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 05:00:21.692605 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:00:21.692605 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 475/942] 10.222.0.77 () {56 vars in 1111 bytes} [Tue Aug 30 05:00:21 2022] GET /identity/v3/projects/service => generated 88 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:00:21.698510 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71504100-71e6-4663-b79a-08c2ebaff157 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:21.753488 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:21.808454 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71504100-71e6-4663-b79a-08c2ebaff157 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:21.844712 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37e56400-1a0e-416a-aed1-5a4a8a250c4e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:21.845243 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37e56400-1a0e-416a-aed1-5a4a8a250c4e None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:21.845519 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37e56400-1a0e-416a-aed1-5a4a8a250c4e None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:21.845788 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37e56400-1a0e-416a-aed1-5a4a8a250c4e None admin] PATH_INFO: `/v3/projects` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:21.846472 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37e56400-1a0e-416a-aed1-5a4a8a250c4e None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:21.846770 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37e56400-1a0e-416a-aed1-5a4a8a250c4e None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:21.848222 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37e56400-1a0e-416a-aed1-5a4a8a250c4e None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:21.866054 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 468/943] 10.222.0.77 () {56 vars in 1148 bytes} [Tue Aug 30 05:00:21 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:21.867640 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcf95020-1af8-46d6-9ecc-34140ed5942e None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:21.935942 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:21.977243 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcf95020-1af8-46d6-9ecc-34140ed5942e None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:22.014197 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60f1822d-34f6-41d5-8387-1d353ff3ce34 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:22.014870 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60f1822d-34f6-41d5-8387-1d353ff3ce34 None admin] REQUEST_METHOD: `PUT` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:22.015154 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60f1822d-34f6-41d5-8387-1d353ff3ce34 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:22.015424 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60f1822d-34f6-41d5-8387-1d353ff3ce34 None admin] PATH_INFO: `/v3/projects/e9919141e3b947ab8d91b55bba4a7de6/users/fe51dbec91624ed1ab00b441e29a3dd5/roles/2d3c2095a28d4f4f8a8328c4b61b0015` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:22.017981 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60f1822d-34f6-41d5-8387-1d353ff3ce34 None admin] RBAC: Authorizing `identity:create_grant(project_id=e9919141e3b947ab8d91b55bba4a7de6, user_id=fe51dbec91624ed1ab00b441e29a3dd5, role_id=2d3c2095a28d4f4f8a8328c4b61b0015)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:22.019877 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60f1822d-34f6-41d5-8387-1d353ff3ce34 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:22.036921 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 476/944] 10.222.0.77 () {58 vars in 1439 bytes} [Tue Aug 30 05:00:21 2022] PUT /identity/v3/projects/e9919141e3b947ab8d91b55bba4a7de6/users/fe51dbec91624ed1ab00b441e29a3dd5/roles/2d3c2095a28d4f4f8a8328c4b61b0015 => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:00:22.123261 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edc71dd3-87bf-4ad1-a80b-49d8dba21530 None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:22.123775 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edc71dd3-87bf-4ad1-a80b-49d8dba21530 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:22.123775 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edc71dd3-87bf-4ad1-a80b-49d8dba21530 None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:22.125395 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 469/945] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:00:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:00:22.130908 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e729ff9d-521e-420d-9776-aabc6f9377b4 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:22.131198 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e729ff9d-521e-420d-9776-aabc6f9377b4 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:22.132055 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e729ff9d-521e-420d-9776-aabc6f9377b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:22.152383 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-e729ff9d-521e-420d-9776-aabc6f9377b4 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:00:22.153615 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-e729ff9d-521e-420d-9776-aabc6f9377b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:00:22.227423 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 477/946] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:00:22 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 05:00:22.504023 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edc71dd3-87bf-4ad1-a80b-49d8dba21530 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:22.581598 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a049115f-6233-40a6-8b2d-296525be9a24 None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:22.581928 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a049115f-6233-40a6-8b2d-296525be9a24 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:22.582255 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a049115f-6233-40a6-8b2d-296525be9a24 None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:22.586947 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 478/947] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:00:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:00:22.589240 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df128b96-386e-4a6c-ad64-7b314ac7c4e4 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:22.589520 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df128b96-386e-4a6c-ad64-7b314ac7c4e4 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:22.589788 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df128b96-386e-4a6c-ad64-7b314ac7c4e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:22.609737 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edc71dd3-87bf-4ad1-a80b-49d8dba21530 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:22.609939 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-df128b96-386e-4a6c-ad64-7b314ac7c4e4 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:00:22.611537 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-df128b96-386e-4a6c-ad64-7b314ac7c4e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:00:22.640938 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:22.647459 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcd4ecad-52b1-46cc-b155-6a8689a32f23 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:22.648081 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd4ecad-52b1-46cc-b155-6a8689a32f23 None admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:22.648352 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd4ecad-52b1-46cc-b155-6a8689a32f23 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:22.648629 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd4ecad-52b1-46cc-b155-6a8689a32f23 None admin] PATH_INFO: `/v3/projects` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:22.649294 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcd4ecad-52b1-46cc-b155-6a8689a32f23 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:22.650736 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcd4ecad-52b1-46cc-b155-6a8689a32f23 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:22.653376 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: WARNING keystone.server.flask.common [None req-bcd4ecad-52b1-46cc-b155-6a8689a32f23 None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. Aug 30 05:00:22.667576 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:22.672611 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.sql.core [None req-bcd4ecad-52b1-46cc-b155-6a8689a32f23 None admin] Conflict project: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Aug 30 05:00:22.672611 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Aug 30 05:00:22.672611 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [parameters: {'id': '4b9f84a946ae4ca38697df6cc797fce2', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Aug 30 05:00:22.672611 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81624) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Aug 30 05:00:22.682113 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-bcd4ecad-52b1-46cc-b155-6a8689a32f23 None admin] Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 4b9f84a946ae4ca38697df6cc797fce2.: keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 4b9f84a946ae4ca38697df6cc797fce2. Aug 30 05:00:22.682113 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:22.682113 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Aug 30 05:00:22.682113 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application self.dialect.do_execute( Aug 30 05:00:22.682113 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Aug 30 05:00:22.682113 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Aug 30 05:00:22.682113 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Aug 30 05:00:22.682113 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application result = self._query(query) Aug 30 05:00:22.682113 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Aug 30 05:00:22.682113 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application conn.query(q) Aug 30 05:00:22.682113 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Aug 30 05:00:22.682113 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Aug 30 05:00:22.682113 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Aug 30 05:00:22.683140 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application result.read() Aug 30 05:00:22.683140 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Aug 30 05:00:22.683140 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Aug 30 05:00:22.683140 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Aug 30 05:00:22.683140 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application packet.raise_for_error() Aug 30 05:00:22.683140 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Aug 30 05:00:22.683140 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Aug 30 05:00:22.683140 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Aug 30 05:00:22.683140 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Aug 30 05:00:22.683140 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Aug 30 05:00:22.683140 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:00:22.683140 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Aug 30 05:00:22.683140 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:00:22.683140 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:22.683140 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Aug 30 05:00:22.683140 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return method(*args, **kwargs) Aug 30 05:00:22.683140 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 241, in create_project Aug 30 05:00:22.683140 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return project_ref.to_dict() Aug 30 05:00:22.684578 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Aug 30 05:00:22.684578 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application next(self.gen) Aug 30 05:00:22.684578 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Aug 30 05:00:22.684578 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application yield resource Aug 30 05:00:22.684578 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Aug 30 05:00:22.684578 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application next(self.gen) Aug 30 05:00:22.684578 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Aug 30 05:00:22.684578 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application self.session.rollback() Aug 30 05:00:22.684578 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Aug 30 05:00:22.684578 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application self.force_reraise() Aug 30 05:00:22.684578 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Aug 30 05:00:22.684578 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise self.value Aug 30 05:00:22.684578 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Aug 30 05:00:22.684578 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Aug 30 05:00:22.684578 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Aug 30 05:00:22.684578 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application session.commit() Aug 30 05:00:22.684578 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Aug 30 05:00:22.685703 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Aug 30 05:00:22.685703 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Aug 30 05:00:22.685703 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application self._prepare_impl() Aug 30 05:00:22.685703 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Aug 30 05:00:22.685703 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application self.session.flush() Aug 30 05:00:22.685703 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Aug 30 05:00:22.685703 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application self._flush(objects) Aug 30 05:00:22.685703 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Aug 30 05:00:22.685703 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Aug 30 05:00:22.685703 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Aug 30 05:00:22.685703 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application compat.raise_( Aug 30 05:00:22.685703 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Aug 30 05:00:22.685703 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception Aug 30 05:00:22.685703 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Aug 30 05:00:22.685703 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application flush_context.execute() Aug 30 05:00:22.685703 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Aug 30 05:00:22.685703 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rec.execute(self) Aug 30 05:00:22.686865 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Aug 30 05:00:22.686865 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Aug 30 05:00:22.686865 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Aug 30 05:00:22.686865 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application _emit_insert_statements( Aug 30 05:00:22.686865 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Aug 30 05:00:22.686865 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application c = connection._execute_20( Aug 30 05:00:22.686865 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Aug 30 05:00:22.686865 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Aug 30 05:00:22.686865 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Aug 30 05:00:22.686865 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Aug 30 05:00:22.686865 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Aug 30 05:00:22.686865 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ret = self._execute_context( Aug 30 05:00:22.686865 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Aug 30 05:00:22.686865 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Aug 30 05:00:22.686865 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Aug 30 05:00:22.688118 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Aug 30 05:00:22.688118 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Aug 30 05:00:22.688118 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception Aug 30 05:00:22.688118 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Aug 30 05:00:22.688118 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application self.dialect.do_execute( Aug 30 05:00:22.688118 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Aug 30 05:00:22.688118 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Aug 30 05:00:22.688118 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Aug 30 05:00:22.688118 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application result = self._query(query) Aug 30 05:00:22.688118 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Aug 30 05:00:22.688118 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application conn.query(q) Aug 30 05:00:22.688118 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Aug 30 05:00:22.688118 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Aug 30 05:00:22.688118 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Aug 30 05:00:22.688118 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application result.read() Aug 30 05:00:22.688118 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Aug 30 05:00:22.688118 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Aug 30 05:00:22.690204 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Aug 30 05:00:22.690204 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application packet.raise_for_error() Aug 30 05:00:22.690204 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Aug 30 05:00:22.690204 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Aug 30 05:00:22.690204 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Aug 30 05:00:22.690204 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Aug 30 05:00:22.690204 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Aug 30 05:00:22.690204 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Aug 30 05:00:22.690204 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [parameters: {'id': '4b9f84a946ae4ca38697df6cc797fce2', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Aug 30 05:00:22.690204 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Aug 30 05:00:22.690204 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:00:22.690204 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 05:00:22.690204 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:00:22.690204 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:22.690204 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 216, in create_project Aug 30 05:00:22.691909 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ret = self.driver.create_project(project_id, project) Aug 30 05:00:22.691909 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Aug 30 05:00:22.691909 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Aug 30 05:00:22.691909 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - Duplicate entry found with name service at domain ID default. Aug 30 05:00:22.691909 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:00:22.691909 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 05:00:22.691909 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:00:22.691909 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:22.691909 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:00:22.691909 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:00:22.691909 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:00:22.691909 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:00:22.691909 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:00:22.691909 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:00:22.691909 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:00:22.691909 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:00:22.691909 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:00:22.694403 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:00:22.694403 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 176, in post Aug 30 05:00:22.694403 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ref = PROVIDERS.resource_api.create_project( Aug 30 05:00:22.694403 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:00:22.694403 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:00:22.694403 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 218, in create_project Aug 30 05:00:22.694403 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.Conflict( Aug 30 05:00:22.694403 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 4b9f84a946ae4ca38697df6cc797fce2. Aug 30 05:00:22.694403 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:00:22.694403 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 470/948] 10.222.0.77 () {60 vars in 1140 bytes} [Tue Aug 30 05:00:22 2022] POST /identity/v3/projects => generated 262 bytes in 180 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:00:22.694403 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcd4ecad-52b1-46cc-b155-6a8689a32f23 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:22.707015 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 479/949] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:00:22 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 05:00:22.798961 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcd4ecad-52b1-46cc-b155-6a8689a32f23 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:22.834395 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d84600c7-a6fc-42a0-b98d-85b3b6639925 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:22.835063 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d84600c7-a6fc-42a0-b98d-85b3b6639925 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:22.835344 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d84600c7-a6fc-42a0-b98d-85b3b6639925 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:22.835612 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d84600c7-a6fc-42a0-b98d-85b3b6639925 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:22.842390 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d84600c7-a6fc-42a0-b98d-85b3b6639925 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:22.844064 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d84600c7-a6fc-42a0-b98d-85b3b6639925 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:22.856077 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-d84600c7-a6fc-42a0-b98d-85b3b6639925 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 05:00:22.856077 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:22.856077 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:00:22.856077 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:00:22.856077 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:00:22.856077 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:00:22.856077 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:00:22.856077 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:00:22.856077 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:00:22.856077 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:00:22.856077 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:00:22.856077 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:00:22.856077 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 05:00:22.856077 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 05:00:22.856077 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 05:00:22.856951 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 05:00:22.856951 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:00:22.856951 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:00:22.856951 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:00:22.856951 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:00:22.856951 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:00:22.856951 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:00:22.856951 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:00:22.856951 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 05:00:22.856951 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:00:22.856951 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:00:22.856951 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:00:22.856951 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:00:22.856951 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:00:22.856951 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:00:22.856951 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:00:22.856951 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:00:22.858434 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 05:00:22.858434 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 05:00:22.858434 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 05:00:22.858434 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 05:00:22.858434 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 05:00:22.858434 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 05:00:22.858434 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 05:00:22.858434 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:00:22.858434 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 471/950] 10.222.0.77 () {56 vars in 1111 bytes} [Tue Aug 30 05:00:22 2022] GET /identity/v3/projects/service => generated 88 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:00:22.862961 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df128b96-386e-4a6c-ad64-7b314ac7c4e4 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:22.961610 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df128b96-386e-4a6c-ad64-7b314ac7c4e4 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:22.996400 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daa91485-66dd-4637-9981-c26402c1499f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:22.997046 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa91485-66dd-4637-9981-c26402c1499f None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:22.997319 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa91485-66dd-4637-9981-c26402c1499f None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:22.997589 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa91485-66dd-4637-9981-c26402c1499f None admin] PATH_INFO: `/v3/projects` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:22.998289 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daa91485-66dd-4637-9981-c26402c1499f None admin] RBAC: Adding query filter params (name=service) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:22.998586 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daa91485-66dd-4637-9981-c26402c1499f None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:23.000060 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daa91485-66dd-4637-9981-c26402c1499f None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:23.017003 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 480/951] 10.222.0.77 () {56 vars in 1112 bytes} [Tue Aug 30 05:00:22 2022] GET /identity/v3/projects?name=service => generated 411 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:23.025107 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d84600c7-a6fc-42a0-b98d-85b3b6639925 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:23.056026 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:23.149436 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d84600c7-a6fc-42a0-b98d-85b3b6639925 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:23.190207 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea9065cd-5c1f-4488-afe3-6012acb35eb3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:23.191037 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9065cd-5c1f-4488-afe3-6012acb35eb3 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:23.191437 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9065cd-5c1f-4488-afe3-6012acb35eb3 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:23.191849 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9065cd-5c1f-4488-afe3-6012acb35eb3 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:23.192499 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea9065cd-5c1f-4488-afe3-6012acb35eb3 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:23.194039 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea9065cd-5c1f-4488-afe3-6012acb35eb3 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:23.206064 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-ea9065cd-5c1f-4488-afe3-6012acb35eb3 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 30 05:00:23.206064 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:23.206064 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 30 05:00:23.206064 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 30 05:00:23.206064 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 30 05:00:23.206064 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 30 05:00:23.206064 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 30 05:00:23.206064 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 05:00:23.206064 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 30 05:00:23.206064 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:00:23.206064 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 05:00:23.206064 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:00:23.206064 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:23.206064 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:00:23.206064 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:00:23.206064 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:00:23.206064 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:00:23.207474 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:00:23.207474 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:00:23.207474 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:00:23.207474 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:00:23.207474 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:00:23.207474 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:00:23.207474 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 30 05:00:23.207474 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 30 05:00:23.207474 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 30 05:00:23.207474 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 30 05:00:23.207474 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:00:23.207474 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:00:23.207474 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:00:23.207474 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:00:23.207474 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:00:23.207474 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:00:23.208485 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:00:23.208485 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 05:00:23.208485 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:00:23.208485 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:00:23.208485 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:00:23.208485 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:00:23.208485 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:00:23.208485 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:00:23.208485 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:00:23.208485 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:00:23.208485 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 30 05:00:23.208485 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 05:00:23.208485 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 30 05:00:23.208485 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:00:23.210950 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 472/952] 10.222.0.77 () {56 vars in 1126 bytes} [Tue Aug 30 05:00:23 2022] GET /identity/v3/services/object-store => generated 93 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:00:23.216854 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daa91485-66dd-4637-9981-c26402c1499f None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:23.268791 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:23.326661 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daa91485-66dd-4637-9981-c26402c1499f None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:23.362121 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5685e06-271c-4ef5-978e-5135fc313e3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:23.362911 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5685e06-271c-4ef5-978e-5135fc313e3b None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:23.363287 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5685e06-271c-4ef5-978e-5135fc313e3b None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:23.363650 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5685e06-271c-4ef5-978e-5135fc313e3b None admin] PATH_INFO: `/v3/services` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:23.364365 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5685e06-271c-4ef5-978e-5135fc313e3b None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:23.364760 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5685e06-271c-4ef5-978e-5135fc313e3b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:23.366785 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5685e06-271c-4ef5-978e-5135fc313e3b None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:23.372083 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 481/953] 10.222.0.77 () {56 vars in 1122 bytes} [Tue Aug 30 05:00:23 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:23.377882 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea9065cd-5c1f-4488-afe3-6012acb35eb3 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:23.487312 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:23.501558 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea9065cd-5c1f-4488-afe3-6012acb35eb3 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:23.538714 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-864ec274-1670-4e06-a385-e5f95225b76e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:23.539336 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-864ec274-1670-4e06-a385-e5f95225b76e None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:23.539608 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-864ec274-1670-4e06-a385-e5f95225b76e None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:23.539875 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-864ec274-1670-4e06-a385-e5f95225b76e None admin] PATH_INFO: `/v3/services` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:23.540498 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-864ec274-1670-4e06-a385-e5f95225b76e None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:23.540793 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-864ec274-1670-4e06-a385-e5f95225b76e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:23.542154 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-864ec274-1670-4e06-a385-e5f95225b76e None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:23.550462 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 473/954] 10.222.0.77 () {56 vars in 1122 bytes} [Tue Aug 30 05:00:23 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:23.556857 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5685e06-271c-4ef5-978e-5135fc313e3b None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:23.620356 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d555fa1-078b-4f38-9a25-e2ff0f0922a1 None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:23.620715 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d555fa1-078b-4f38-9a25-e2ff0f0922a1 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:23.621081 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d555fa1-078b-4f38-9a25-e2ff0f0922a1 None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:23.622921 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 474/955] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:00:23 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:00:23.627930 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f104e98c-0a92-4a23-803b-f115550e3266 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:23.628311 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f104e98c-0a92-4a23-803b-f115550e3266 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:23.628682 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f104e98c-0a92-4a23-803b-f115550e3266 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:23.647776 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-f104e98c-0a92-4a23-803b-f115550e3266 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:00:23.649445 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-f104e98c-0a92-4a23-803b-f115550e3266 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:00:23.669716 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5685e06-271c-4ef5-978e-5135fc313e3b None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:23.682861 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:23.716972 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b4f1616-5368-4005-bb87-950b3b3e8746 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:23.718114 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b4f1616-5368-4005-bb87-950b3b3e8746 None admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:23.719013 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b4f1616-5368-4005-bb87-950b3b3e8746 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:23.719435 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b4f1616-5368-4005-bb87-950b3b3e8746 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:23.720584 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b4f1616-5368-4005-bb87-950b3b3e8746 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:23.722325 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b4f1616-5368-4005-bb87-950b3b3e8746 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:23.731324 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 475/956] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:00:23 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 30 05:00:23.750988 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 482/957] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:00:23 2022] POST /identity/v3/endpoints => generated 349 bytes in 191 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:00:23.997655 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f104e98c-0a92-4a23-803b-f115550e3266 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:24.041859 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d68fb41-b8f7-44e8-8d47-180aadd3656d None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:24.042403 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d68fb41-b8f7-44e8-8d47-180aadd3656d None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:24.042403 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d68fb41-b8f7-44e8-8d47-180aadd3656d None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:24.044016 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 483/958] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:00:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:00:24.048829 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-318c49d0-5d76-4232-a12d-e9cc789e1cec None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:24.049111 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-318c49d0-5d76-4232-a12d-e9cc789e1cec None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:24.049385 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-318c49d0-5d76-4232-a12d-e9cc789e1cec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:24.067343 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-318c49d0-5d76-4232-a12d-e9cc789e1cec None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:00:24.068532 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-318c49d0-5d76-4232-a12d-e9cc789e1cec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:00:24.091650 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:24.111863 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f104e98c-0a92-4a23-803b-f115550e3266 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:24.155264 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-496005e2-9e3c-4c40-a860-6b0cd20bde6b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:24.155908 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-496005e2-9e3c-4c40-a860-6b0cd20bde6b None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:24.156154 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-496005e2-9e3c-4c40-a860-6b0cd20bde6b None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:24.156679 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-496005e2-9e3c-4c40-a860-6b0cd20bde6b None admin] PATH_INFO: `/v3/services/compute` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:24.156993 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-496005e2-9e3c-4c40-a860-6b0cd20bde6b None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:24.158359 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-496005e2-9e3c-4c40-a860-6b0cd20bde6b None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:24.165989 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 484/959] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:00:24 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 30 05:00:24.169740 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-496005e2-9e3c-4c40-a860-6b0cd20bde6b None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Aug 30 05:00:24.169740 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:24.169740 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 30 05:00:24.169740 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 30 05:00:24.169740 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 30 05:00:24.169740 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 30 05:00:24.169740 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 30 05:00:24.169740 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 05:00:24.169740 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Aug 30 05:00:24.169740 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:00:24.169740 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 05:00:24.169740 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:00:24.169740 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:24.169740 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:00:24.169740 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:00:24.169740 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:00:24.169740 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:00:24.171008 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:00:24.171008 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:00:24.171008 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:00:24.171008 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:00:24.171008 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:00:24.171008 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:00:24.171008 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 30 05:00:24.171008 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 30 05:00:24.171008 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 30 05:00:24.171008 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 30 05:00:24.171008 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:00:24.171008 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:00:24.171008 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:00:24.171008 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:00:24.171008 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:00:24.171008 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:00:24.172084 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:00:24.172084 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 05:00:24.172084 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:00:24.172084 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:00:24.172084 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:00:24.172084 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:00:24.172084 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:00:24.172084 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:00:24.172084 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:00:24.172084 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:00:24.172084 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 30 05:00:24.172084 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 05:00:24.172084 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Aug 30 05:00:24.172084 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:00:24.172084 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 476/960] 10.222.0.77 () {56 vars in 1111 bytes} [Tue Aug 30 05:00:23 2022] GET /identity/v3/services/compute => generated 88 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:00:24.179851 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-318c49d0-5d76-4232-a12d-e9cc789e1cec None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:24.281842 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:24.289709 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-318c49d0-5d76-4232-a12d-e9cc789e1cec None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:24.326268 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fd839dd-e7e9-4083-b508-93f213211f58 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:24.326899 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd839dd-e7e9-4083-b508-93f213211f58 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:24.327179 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd839dd-e7e9-4083-b508-93f213211f58 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:24.327448 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd839dd-e7e9-4083-b508-93f213211f58 None admin] PATH_INFO: `/v3/services` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:24.328068 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fd839dd-e7e9-4083-b508-93f213211f58 None admin] RBAC: Adding query filter params (name=compute) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:24.328363 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fd839dd-e7e9-4083-b508-93f213211f58 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:24.329722 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fd839dd-e7e9-4083-b508-93f213211f58 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:24.335485 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 485/961] 10.222.0.77 () {56 vars in 1112 bytes} [Tue Aug 30 05:00:24 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:24.342619 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-496005e2-9e3c-4c40-a860-6b0cd20bde6b None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:24.445687 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fd839dd-e7e9-4083-b508-93f213211f58 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:24.452131 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-496005e2-9e3c-4c40-a860-6b0cd20bde6b None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:24.487409 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f78d7af-779e-4ad0-b4b1-eaf359aeb403 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:24.488025 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f78d7af-779e-4ad0-b4b1-eaf359aeb403 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:24.488299 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f78d7af-779e-4ad0-b4b1-eaf359aeb403 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:24.488567 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f78d7af-779e-4ad0-b4b1-eaf359aeb403 None admin] PATH_INFO: `/v3/services` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:24.489188 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f78d7af-779e-4ad0-b4b1-eaf359aeb403 None admin] RBAC: Adding query filter params (type=compute) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:24.489487 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f78d7af-779e-4ad0-b4b1-eaf359aeb403 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:24.491376 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f78d7af-779e-4ad0-b4b1-eaf359aeb403 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:24.496235 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 477/962] 10.222.0.77 () {56 vars in 1112 bytes} [Tue Aug 30 05:00:24 2022] GET /identity/v3/services?type=compute => generated 124 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:24.575276 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fd839dd-e7e9-4083-b508-93f213211f58 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:24.612711 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b35b48fb-76ae-4f88-a082-8f11925caa8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:24.613356 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b35b48fb-76ae-4f88-a082-8f11925caa8a None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:24.613634 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b35b48fb-76ae-4f88-a082-8f11925caa8a None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:24.613903 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b35b48fb-76ae-4f88-a082-8f11925caa8a None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:24.626059 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b35b48fb-76ae-4f88-a082-8f11925caa8a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 05:00:24.631010 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-b35b48fb-76ae-4f88-a082-8f11925caa8a None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 30 05:00:24.631010 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:24.631010 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:00:24.631010 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:00:24.631010 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:00:24.631010 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:00:24.631010 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:00:24.631010 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:00:24.631010 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:00:24.631010 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:00:24.631010 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:00:24.631010 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:00:24.631010 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 05:00:24.631010 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 05:00:24.631010 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 05:00:24.631010 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 05:00:24.632557 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 05:00:24.632557 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 05:00:24.632557 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 05:00:24.632557 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 05:00:24.632557 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:00:24.632557 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:00:24.632557 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:00:24.632557 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:00:24.632557 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:00:24.632557 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:00:24.632557 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:00:24.632557 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 05:00:24.632557 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:00:24.632557 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:00:24.632557 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:00:24.632557 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:00:24.633413 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:00:24.633413 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:00:24.633413 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:00:24.633413 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:00:24.633413 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 05:00:24.633413 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 05:00:24.633413 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 05:00:24.633413 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 05:00:24.633413 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 05:00:24.633413 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 05:00:24.633413 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 30 05:00:24.633413 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:00:24.633413 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 486/963] 10.222.0.77 () {56 vars in 1120 bytes} [Tue Aug 30 05:00:24 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 05:00:24.639334 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f78d7af-779e-4ad0-b4b1-eaf359aeb403 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:24.649583 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:24.746554 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f78d7af-779e-4ad0-b4b1-eaf359aeb403 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:24.781649 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e38d3a0-92ac-4790-a1ba-c868b70151d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:24.782265 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e38d3a0-92ac-4790-a1ba-c868b70151d3 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:24.782543 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e38d3a0-92ac-4790-a1ba-c868b70151d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:24.782811 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e38d3a0-92ac-4790-a1ba-c868b70151d3 None admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:24.783497 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e38d3a0-92ac-4790-a1ba-c868b70151d3 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:24.783798 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e38d3a0-92ac-4790-a1ba-c868b70151d3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:24.785239 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e38d3a0-92ac-4790-a1ba-c868b70151d3 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:24.794799 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 478/964] 10.222.0.77 () {56 vars in 1115 bytes} [Tue Aug 30 05:00:24 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:24.800329 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b35b48fb-76ae-4f88-a082-8f11925caa8a None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:24.822101 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:24.906223 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b35b48fb-76ae-4f88-a082-8f11925caa8a None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:24.944456 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4644b96-5faf-40f3-864a-ab53116c13d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:24.945430 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4644b96-5faf-40f3-864a-ab53116c13d2 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:24.945908 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4644b96-5faf-40f3-864a-ab53116c13d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:24.946394 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4644b96-5faf-40f3-864a-ab53116c13d2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:24.956928 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4644b96-5faf-40f3-864a-ab53116c13d2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:24.958612 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4644b96-5faf-40f3-864a-ab53116c13d2 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:24.973611 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-d4644b96-5faf-40f3-864a-ab53116c13d2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 05:00:24.973611 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:24.973611 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:00:24.973611 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:00:24.973611 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:00:24.973611 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:00:24.973611 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:00:24.973611 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:00:24.973611 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:00:24.973611 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:00:24.973611 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:00:24.973611 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:00:24.973611 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 05:00:24.973611 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 05:00:24.973611 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 05:00:24.973611 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 05:00:24.974600 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:00:24.974600 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:00:24.974600 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:00:24.974600 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:00:24.974600 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:00:24.974600 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:00:24.974600 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:00:24.974600 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 05:00:24.974600 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:00:24.974600 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:00:24.974600 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:00:24.974600 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:00:24.974600 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:00:24.974600 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:00:24.974600 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:00:24.974600 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:00:24.974600 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 05:00:24.976047 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 05:00:24.976047 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 05:00:24.976047 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:00:24.978369 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 487/965] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 05:00:24 2022] GET /identity/v3/domains/Default => generated 87 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:00:24.983937 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e38d3a0-92ac-4790-a1ba-c868b70151d3 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:25.057966 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:25.094072 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e38d3a0-92ac-4790-a1ba-c868b70151d3 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:25.129083 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f85b9fe-6bcf-4c7a-b7c5-2fa54bcd847b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:25.130091 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f85b9fe-6bcf-4c7a-b7c5-2fa54bcd847b None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:25.130572 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f85b9fe-6bcf-4c7a-b7c5-2fa54bcd847b None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:25.131067 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f85b9fe-6bcf-4c7a-b7c5-2fa54bcd847b None admin] PATH_INFO: `/v3/domains` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:25.132421 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f85b9fe-6bcf-4c7a-b7c5-2fa54bcd847b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:25.132917 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f85b9fe-6bcf-4c7a-b7c5-2fa54bcd847b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:25.135317 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f85b9fe-6bcf-4c7a-b7c5-2fa54bcd847b None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:25.135894 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8306786b-9061-4b05-b964-9a0a745b23e5 None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:25.136713 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8306786b-9061-4b05-b964-9a0a745b23e5 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:25.137351 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8306786b-9061-4b05-b964-9a0a745b23e5 None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:25.139687 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 488/966] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:00:25 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 05:00:25.144197 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8ad0112-0b90-4f58-aef6-7370dc8fb970 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:25.144476 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8ad0112-0b90-4f58-aef6-7370dc8fb970 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:25.144742 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8ad0112-0b90-4f58-aef6-7370dc8fb970 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:25.153518 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 479/967] 10.222.0.77 () {56 vars in 1109 bytes} [Tue Aug 30 05:00:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:25.159356 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f85b9fe-6bcf-4c7a-b7c5-2fa54bcd847b None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:25.168758 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-c8ad0112-0b90-4f58-aef6-7370dc8fb970 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:00:25.170073 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-c8ad0112-0b90-4f58-aef6-7370dc8fb970 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:00:25.242422 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:25.247604 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 489/968] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:00:25 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 30 05:00:25.289073 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f85b9fe-6bcf-4c7a-b7c5-2fa54bcd847b None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:25.316950 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:25.326192 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6502d921-e69c-426e-87ab-d63546c4d3ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:25.326838 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6502d921-e69c-426e-87ab-d63546c4d3ea None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:25.327125 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6502d921-e69c-426e-87ab-d63546c4d3ea None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:25.327394 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6502d921-e69c-426e-87ab-d63546c4d3ea None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:25.334840 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6502d921-e69c-426e-87ab-d63546c4d3ea None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:25.336279 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6502d921-e69c-426e-87ab-d63546c4d3ea None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:25.348100 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-6502d921-e69c-426e-87ab-d63546c4d3ea None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 30 05:00:25.348100 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:25.348100 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:00:25.348100 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:00:25.348100 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:00:25.348100 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:00:25.348100 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:00:25.348100 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:00:25.348100 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:00:25.348100 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:00:25.348100 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:00:25.348100 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:00:25.348100 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 05:00:25.348100 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 05:00:25.348100 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 05:00:25.348100 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 05:00:25.349034 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:00:25.349034 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:00:25.349034 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 05:00:25.349034 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 05:00:25.349034 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 05:00:25.349034 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 05:00:25.349034 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:00:25.349034 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:00:25.349034 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:00:25.349034 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:00:25.349034 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:00:25.349034 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 05:00:25.349034 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:00:25.349034 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:00:25.349034 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:00:25.349034 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:00:25.349034 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:00:25.349034 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:00:25.349980 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:00:25.349980 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:00:25.349980 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 05:00:25.349980 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 05:00:25.349980 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 05:00:25.349980 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 05:00:25.349980 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 05:00:25.349980 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 05:00:25.349980 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 30 05:00:25.349980 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:00:25.349980 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 480/969] 10.222.0.77 () {56 vars in 1117 bytes} [Tue Aug 30 05:00:25 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:00:25.355626 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8ad0112-0b90-4f58-aef6-7370dc8fb970 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:25.466119 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8ad0112-0b90-4f58-aef6-7370dc8fb970 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:25.500690 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88f510bc-3bfb-41d5-9104-f41459cba07e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:25.501289 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88f510bc-3bfb-41d5-9104-f41459cba07e None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:25.501565 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88f510bc-3bfb-41d5-9104-f41459cba07e None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:25.501831 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88f510bc-3bfb-41d5-9104-f41459cba07e None admin] PATH_INFO: `/v3/users` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:25.502551 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88f510bc-3bfb-41d5-9104-f41459cba07e None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:25.502882 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88f510bc-3bfb-41d5-9104-f41459cba07e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:25.504311 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88f510bc-3bfb-41d5-9104-f41459cba07e None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:25.515728 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6502d921-e69c-426e-87ab-d63546c4d3ea None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:25.523326 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 490/970] 10.222.0.77 () {56 vars in 1149 bytes} [Tue Aug 30 05:00:25 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:25.531849 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88f510bc-3bfb-41d5-9104-f41459cba07e None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:25.607356 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:25.651601 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6502d921-e69c-426e-87ab-d63546c4d3ea None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:25.659890 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88f510bc-3bfb-41d5-9104-f41459cba07e None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:25.700148 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca7ae634-5ac7-4b78-88f5-733f8f8003dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:25.701694 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca7ae634-5ac7-4b78-88f5-733f8f8003dd None admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:25.702224 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca7ae634-5ac7-4b78-88f5-733f8f8003dd None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:25.702645 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca7ae634-5ac7-4b78-88f5-733f8f8003dd None admin] PATH_INFO: `/v3/users` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:25.703451 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca7ae634-5ac7-4b78-88f5-733f8f8003dd None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:25.705312 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca7ae634-5ac7-4b78-88f5-733f8f8003dd None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:25.705447 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b539b07-d65f-41cc-b6c5-7db66636bfd8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:25.705915 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b539b07-d65f-41cc-b6c5-7db66636bfd8 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:25.706188 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b539b07-d65f-41cc-b6c5-7db66636bfd8 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:25.706452 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b539b07-d65f-41cc-b6c5-7db66636bfd8 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:25.726344 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.sql.core [None req-ca7ae634-5ac7-4b78-88f5-733f8f8003dd None admin] Conflict user: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Aug 30 05:00:25.726344 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Aug 30 05:00:25.726344 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [parameters: {'user_id': '3b920fc205444d458c8d172ac11496f9', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Aug 30 05:00:25.726344 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81624) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Aug 30 05:00:25.734924 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-ca7ae634-5ac7-4b78-88f5-733f8f8003dd None admin] Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default.: keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Aug 30 05:00:25.734924 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:25.734924 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Aug 30 05:00:25.734924 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application self.dialect.do_execute( Aug 30 05:00:25.734924 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Aug 30 05:00:25.734924 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Aug 30 05:00:25.734924 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Aug 30 05:00:25.734924 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application result = self._query(query) Aug 30 05:00:25.734924 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Aug 30 05:00:25.734924 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application conn.query(q) Aug 30 05:00:25.734924 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Aug 30 05:00:25.734924 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Aug 30 05:00:25.734924 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Aug 30 05:00:25.734924 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application result.read() Aug 30 05:00:25.734924 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Aug 30 05:00:25.735984 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Aug 30 05:00:25.735984 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Aug 30 05:00:25.735984 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application packet.raise_for_error() Aug 30 05:00:25.735984 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Aug 30 05:00:25.735984 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Aug 30 05:00:25.735984 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Aug 30 05:00:25.735984 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Aug 30 05:00:25.735984 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Aug 30 05:00:25.735984 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:00:25.735984 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Aug 30 05:00:25.735984 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:00:25.735984 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:25.735984 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Aug 30 05:00:25.735984 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return method(*args, **kwargs) Aug 30 05:00:25.735984 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 141, in create_user Aug 30 05:00:25.735984 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return base.filter_user(user_ref.to_dict()) Aug 30 05:00:25.735984 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Aug 30 05:00:25.735984 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application next(self.gen) Aug 30 05:00:25.737088 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Aug 30 05:00:25.737088 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application yield resource Aug 30 05:00:25.737088 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Aug 30 05:00:25.737088 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application next(self.gen) Aug 30 05:00:25.737088 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Aug 30 05:00:25.737088 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application self.session.rollback() Aug 30 05:00:25.737088 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Aug 30 05:00:25.737088 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application self.force_reraise() Aug 30 05:00:25.737088 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Aug 30 05:00:25.737088 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise self.value Aug 30 05:00:25.737088 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Aug 30 05:00:25.737088 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Aug 30 05:00:25.737088 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Aug 30 05:00:25.737088 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application session.commit() Aug 30 05:00:25.737088 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Aug 30 05:00:25.737088 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Aug 30 05:00:25.737088 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Aug 30 05:00:25.738015 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application self._prepare_impl() Aug 30 05:00:25.738015 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Aug 30 05:00:25.738015 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application self.session.flush() Aug 30 05:00:25.738015 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Aug 30 05:00:25.738015 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application self._flush(objects) Aug 30 05:00:25.738015 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Aug 30 05:00:25.738015 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Aug 30 05:00:25.738015 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Aug 30 05:00:25.738015 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application compat.raise_( Aug 30 05:00:25.738015 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Aug 30 05:00:25.738015 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception Aug 30 05:00:25.738015 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Aug 30 05:00:25.738015 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application flush_context.execute() Aug 30 05:00:25.738015 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Aug 30 05:00:25.738015 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rec.execute(self) Aug 30 05:00:25.738015 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Aug 30 05:00:25.738015 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Aug 30 05:00:25.738957 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Aug 30 05:00:25.738957 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application _emit_insert_statements( Aug 30 05:00:25.738957 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1238, in _emit_insert_statements Aug 30 05:00:25.738957 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application result = connection._execute_20( Aug 30 05:00:25.738957 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Aug 30 05:00:25.738957 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Aug 30 05:00:25.738957 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Aug 30 05:00:25.738957 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Aug 30 05:00:25.738957 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Aug 30 05:00:25.738957 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ret = self._execute_context( Aug 30 05:00:25.738957 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Aug 30 05:00:25.738957 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Aug 30 05:00:25.738957 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Aug 30 05:00:25.738957 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Aug 30 05:00:25.738957 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Aug 30 05:00:25.738957 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception Aug 30 05:00:25.739813 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Aug 30 05:00:25.739813 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application self.dialect.do_execute( Aug 30 05:00:25.739813 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Aug 30 05:00:25.739813 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Aug 30 05:00:25.739813 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Aug 30 05:00:25.739813 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application result = self._query(query) Aug 30 05:00:25.739813 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Aug 30 05:00:25.739813 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application conn.query(q) Aug 30 05:00:25.739813 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Aug 30 05:00:25.739813 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Aug 30 05:00:25.739813 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Aug 30 05:00:25.739813 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application result.read() Aug 30 05:00:25.739813 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Aug 30 05:00:25.739813 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Aug 30 05:00:25.739813 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Aug 30 05:00:25.739813 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application packet.raise_for_error() Aug 30 05:00:25.741304 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Aug 30 05:00:25.741304 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Aug 30 05:00:25.741304 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Aug 30 05:00:25.741304 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Aug 30 05:00:25.741304 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Aug 30 05:00:25.741304 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Aug 30 05:00:25.741304 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [parameters: {'user_id': '3b920fc205444d458c8d172ac11496f9', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Aug 30 05:00:25.741304 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Aug 30 05:00:25.741304 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:00:25.741304 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 05:00:25.741304 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:00:25.741304 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:25.741304 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:00:25.741304 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:00:25.741304 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:00:25.742122 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:00:25.742122 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:00:25.742122 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:00:25.742122 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:00:25.742122 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:00:25.742122 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:00:25.742122 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:00:25.742122 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 244, in post Aug 30 05:00:25.742122 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.create_user( Aug 30 05:00:25.742122 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:00:25.742122 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:00:25.742122 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 05:00:25.742122 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 05:00:25.742122 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 05:00:25.742122 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 05:00:25.742122 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1000, in create_user Aug 30 05:00:25.742122 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ref = self._create_user_with_federated_objects(user, driver) Aug 30 05:00:25.743056 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 966, in _create_user_with_federated_objects Aug 30 05:00:25.743056 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application user = driver.create_user(user['id'], user) Aug 30 05:00:25.743056 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Aug 30 05:00:25.743056 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Aug 30 05:00:25.743056 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Aug 30 05:00:25.743056 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:00:25.746866 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 481/971] 10.222.0.77 () {60 vars in 1131 bytes} [Tue Aug 30 05:00:25 2022] POST /identity/v3/users => generated 160 bytes in 225 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:00:25.747898 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca7ae634-5ac7-4b78-88f5-733f8f8003dd None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:25.802941 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b539b07-d65f-41cc-b6c5-7db66636bfd8 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:25.805017 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b539b07-d65f-41cc-b6c5-7db66636bfd8 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:25.807084 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:25.861719 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca7ae634-5ac7-4b78-88f5-733f8f8003dd None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:25.867829 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:25.901562 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dffedb01-1b9e-4ec2-b0c7-73c89c3da002 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:25.902202 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dffedb01-1b9e-4ec2-b0c7-73c89c3da002 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:25.902479 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dffedb01-1b9e-4ec2-b0c7-73c89c3da002 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:25.902749 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dffedb01-1b9e-4ec2-b0c7-73c89c3da002 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:25.910687 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dffedb01-1b9e-4ec2-b0c7-73c89c3da002 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:25.912333 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dffedb01-1b9e-4ec2-b0c7-73c89c3da002 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:25.913522 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 491/972] 10.222.0.77 () {58 vars in 1282 bytes} [Tue Aug 30 05:00:25 2022] GET /identity/v3/auth/tokens => generated 1474 bytes in 382 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Aug 30 05:00:25.915150 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1837ca63-9eb2-45a1-b90b-cc6329047c6c None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:25.915439 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1837ca63-9eb2-45a1-b90b-cc6329047c6c None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:25.915713 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1837ca63-9eb2-45a1-b90b-cc6329047c6c None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:25.917086 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 492/973] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:00:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:00:25.919988 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1837ca63-9eb2-45a1-b90b-cc6329047c6c None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:25.929952 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-dffedb01-1b9e-4ec2-b0c7-73c89c3da002 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Aug 30 05:00:25.929952 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:25.929952 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:00:25.929952 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:00:25.929952 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:00:25.929952 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:00:25.929952 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:00:25.929952 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:00:25.929952 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:00:25.929952 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:00:25.929952 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:00:25.929952 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:00:25.929952 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 05:00:25.929952 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 05:00:25.929952 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 05:00:25.929952 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 05:00:25.931124 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:00:25.931124 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:00:25.931124 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 05:00:25.931124 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 05:00:25.931124 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 05:00:25.931124 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 05:00:25.931124 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:00:25.931124 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:00:25.931124 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:00:25.931124 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:00:25.931124 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:00:25.931124 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 05:00:25.931124 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:00:25.931124 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:00:25.931124 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:00:25.931124 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:00:25.931124 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:00:25.931124 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:00:25.932165 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:00:25.932165 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:00:25.932165 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 05:00:25.932165 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 05:00:25.932165 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 05:00:25.932165 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 05:00:25.932165 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 05:00:25.932165 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 05:00:25.932165 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Aug 30 05:00:25.932165 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:00:25.937522 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 482/974] 10.222.0.77 () {56 vars in 1099 bytes} [Tue Aug 30 05:00:25 2022] GET /identity/v3/users/cinder => generated 84 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:00:25.937522 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-228c2dc8-2ec8-4729-9581-8c23a453f5ba None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:25.937522 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-228c2dc8-2ec8-4729-9581-8c23a453f5ba None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:25.937522 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-228c2dc8-2ec8-4729-9581-8c23a453f5ba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:25.953847 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-228c2dc8-2ec8-4729-9581-8c23a453f5ba None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:00:25.955399 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-228c2dc8-2ec8-4729-9581-8c23a453f5ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:00:26.024499 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 483/975] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:00:25 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 05:00:26.026327 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1837ca63-9eb2-45a1-b90b-cc6329047c6c None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:26.031689 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-228c2dc8-2ec8-4729-9581-8c23a453f5ba None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:26.063418 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-962b014b-b11b-43a1-9035-2424b68f5817 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:26.064039 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-962b014b-b11b-43a1-9035-2424b68f5817 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:26.064316 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-962b014b-b11b-43a1-9035-2424b68f5817 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:26.064586 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-962b014b-b11b-43a1-9035-2424b68f5817 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:26.075732 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-962b014b-b11b-43a1-9035-2424b68f5817 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:26.077179 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-962b014b-b11b-43a1-9035-2424b68f5817 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:26.089037 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:26.092208 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-962b014b-b11b-43a1-9035-2424b68f5817 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 05:00:26.092208 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:26.092208 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:00:26.092208 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:00:26.092208 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:00:26.092208 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:00:26.092208 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:00:26.092208 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:00:26.092208 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:00:26.092208 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:00:26.092208 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:00:26.092208 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:00:26.092208 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 05:00:26.092208 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 05:00:26.092208 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 05:00:26.092208 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 05:00:26.094935 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:00:26.094935 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:00:26.094935 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:00:26.094935 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:00:26.094935 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:00:26.094935 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:00:26.094935 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:00:26.094935 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 05:00:26.094935 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:00:26.094935 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:00:26.094935 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:00:26.094935 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:00:26.094935 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:00:26.094935 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:00:26.094935 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:00:26.094935 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:00:26.094935 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 05:00:26.095818 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 05:00:26.095818 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 05:00:26.095818 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:00:26.098150 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 493/976] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 05:00:25 2022] GET /identity/v3/domains/Default => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:00:26.099050 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-962b014b-b11b-43a1-9035-2424b68f5817 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:26.113886 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:26.160229 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-228c2dc8-2ec8-4729-9581-8c23a453f5ba None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:26.198076 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ca7a843-4713-4f60-afe2-012e5e77a6c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:26.198658 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca7a843-4713-4f60-afe2-012e5e77a6c7 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:26.198916 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca7a843-4713-4f60-afe2-012e5e77a6c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:26.199191 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca7a843-4713-4f60-afe2-012e5e77a6c7 None admin] PATH_INFO: `/v3/users` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:26.199890 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ca7a843-4713-4f60-afe2-012e5e77a6c7 None admin] RBAC: Adding query filter params (name=cinder) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:26.200188 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ca7a843-4713-4f60-afe2-012e5e77a6c7 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:26.201607 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ca7a843-4713-4f60-afe2-012e5e77a6c7 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:26.221053 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-962b014b-b11b-43a1-9035-2424b68f5817 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:26.244692 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 484/977] 10.222.0.77 () {56 vars in 1101 bytes} [Tue Aug 30 05:00:26 2022] GET /identity/v3/users?name=cinder => generated 354 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:26.267533 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07145026-6f5c-4bb5-8a69-fb0547efaaec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:26.268154 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07145026-6f5c-4bb5-8a69-fb0547efaaec None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:26.268430 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07145026-6f5c-4bb5-8a69-fb0547efaaec None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:26.268700 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07145026-6f5c-4bb5-8a69-fb0547efaaec None admin] PATH_INFO: `/v3/domains` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:26.269353 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07145026-6f5c-4bb5-8a69-fb0547efaaec None admin] RBAC: Adding query filter params (name=Default) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:26.269652 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07145026-6f5c-4bb5-8a69-fb0547efaaec None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:26.271094 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07145026-6f5c-4bb5-8a69-fb0547efaaec None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:26.284998 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 494/978] 10.222.0.77 () {56 vars in 1109 bytes} [Tue Aug 30 05:00:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:26.292365 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ca7a843-4713-4f60-afe2-012e5e77a6c7 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:26.297864 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07145026-6f5c-4bb5-8a69-fb0547efaaec None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:26.394115 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:26.415342 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:26.418283 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ca7a843-4713-4f60-afe2-012e5e77a6c7 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:26.437195 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07145026-6f5c-4bb5-8a69-fb0547efaaec None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:26.462798 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d91b5170-fda7-4dec-b741-5a8b5690fad8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:26.463566 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d91b5170-fda7-4dec-b741-5a8b5690fad8 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:26.463916 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d91b5170-fda7-4dec-b741-5a8b5690fad8 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:26.464256 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d91b5170-fda7-4dec-b741-5a8b5690fad8 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:26.473414 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d91b5170-fda7-4dec-b741-5a8b5690fad8 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:26.474961 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d91b5170-fda7-4dec-b741-5a8b5690fad8 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:26.489038 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-217797d7-e314-4da5-bd42-c65bce71b328 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:26.489330 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-d91b5170-fda7-4dec-b741-5a8b5690fad8 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 05:00:26.489330 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:26.489330 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:00:26.489330 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:00:26.489330 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:00:26.489330 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:00:26.489330 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:00:26.489330 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:00:26.489330 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:00:26.489330 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:00:26.489330 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:00:26.489330 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:00:26.489330 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 05:00:26.489330 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 05:00:26.489330 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 05:00:26.490196 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 05:00:26.490196 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:00:26.490196 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:00:26.490196 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:00:26.490196 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:00:26.490196 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:00:26.490196 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:00:26.490196 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:00:26.490196 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 05:00:26.490196 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:00:26.490196 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:00:26.490196 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:00:26.490196 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:00:26.490196 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:00:26.490196 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:00:26.490196 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:00:26.490196 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:00:26.492141 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 05:00:26.492141 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 05:00:26.492141 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 05:00:26.492141 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 05:00:26.492141 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 05:00:26.492141 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 05:00:26.492141 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 05:00:26.492141 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:00:26.492577 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-217797d7-e314-4da5-bd42-c65bce71b328 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:26.492577 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-217797d7-e314-4da5-bd42-c65bce71b328 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:26.492577 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-217797d7-e314-4da5-bd42-c65bce71b328 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:26.492577 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-217797d7-e314-4da5-bd42-c65bce71b328 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:26.492839 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 485/979] 10.222.0.77 () {56 vars in 1111 bytes} [Tue Aug 30 05:00:26 2022] GET /identity/v3/projects/service => generated 88 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:00:26.497680 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d91b5170-fda7-4dec-b741-5a8b5690fad8 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:26.498137 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-217797d7-e314-4da5-bd42-c65bce71b328 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:26.511882 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-217797d7-e314-4da5-bd42-c65bce71b328 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 30 05:00:26.511882 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:26.511882 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 30 05:00:26.511882 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 30 05:00:26.511882 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 30 05:00:26.511882 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 30 05:00:26.511882 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 30 05:00:26.511882 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 05:00:26.511882 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 30 05:00:26.511882 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:00:26.511882 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 05:00:26.511882 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:00:26.511882 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:26.511882 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:00:26.511882 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:00:26.511882 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:00:26.511882 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:00:26.513107 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:00:26.513107 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:00:26.513107 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:00:26.513107 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:00:26.513107 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:00:26.513107 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:00:26.513107 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 30 05:00:26.513107 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 30 05:00:26.513107 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 30 05:00:26.513107 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 30 05:00:26.513107 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:00:26.513107 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:00:26.513107 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:00:26.513107 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:00:26.513107 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:00:26.513107 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:00:26.513979 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:00:26.513979 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 05:00:26.513979 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:00:26.513979 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:00:26.513979 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:00:26.513979 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:00:26.513979 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:00:26.513979 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:00:26.513979 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:00:26.513979 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:00:26.513979 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 30 05:00:26.513979 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 05:00:26.513979 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 30 05:00:26.513979 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:00:26.513979 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 495/980] 10.222.0.77 () {56 vars in 1126 bytes} [Tue Aug 30 05:00:26 2022] GET /identity/v3/services/object-store => generated 93 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:00:26.520246 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-217797d7-e314-4da5-bd42-c65bce71b328 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:26.615476 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d91b5170-fda7-4dec-b741-5a8b5690fad8 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:26.625014 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-217797d7-e314-4da5-bd42-c65bce71b328 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:26.649353 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b442e704-d0fa-4fb5-9fae-d48810205897 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:26.649952 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b442e704-d0fa-4fb5-9fae-d48810205897 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:26.650227 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b442e704-d0fa-4fb5-9fae-d48810205897 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:26.650492 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b442e704-d0fa-4fb5-9fae-d48810205897 None admin] PATH_INFO: `/v3/projects` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:26.651196 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b442e704-d0fa-4fb5-9fae-d48810205897 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:26.651491 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b442e704-d0fa-4fb5-9fae-d48810205897 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:26.652908 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b442e704-d0fa-4fb5-9fae-d48810205897 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:26.658417 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1c00c30-c5c1-4cb0-8bb6-afa1902b6e21 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:26.659046 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1c00c30-c5c1-4cb0-8bb6-afa1902b6e21 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:26.659328 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1c00c30-c5c1-4cb0-8bb6-afa1902b6e21 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:26.659599 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1c00c30-c5c1-4cb0-8bb6-afa1902b6e21 None admin] PATH_INFO: `/v3/services` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:26.660221 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1c00c30-c5c1-4cb0-8bb6-afa1902b6e21 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:26.660526 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1c00c30-c5c1-4cb0-8bb6-afa1902b6e21 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:26.661862 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1c00c30-c5c1-4cb0-8bb6-afa1902b6e21 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:26.663434 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:26.670385 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 486/981] 10.222.0.77 () {56 vars in 1148 bytes} [Tue Aug 30 05:00:26 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 05:00:26.670385 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a4e3f7-0192-4142-8c68-a2198e068154 None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:26.670385 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a4e3f7-0192-4142-8c68-a2198e068154 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:26.670385 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a4e3f7-0192-4142-8c68-a2198e068154 None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:26.670385 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 487/982] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:00:26 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:00:26.671321 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70a4e3f7-0192-4142-8c68-a2198e068154 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:26.672847 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 496/983] 10.222.0.77 () {56 vars in 1122 bytes} [Tue Aug 30 05:00:26 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 05:00:26.674020 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fea8e8b-b3be-406d-85fd-ef724cde8199 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:26.674301 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fea8e8b-b3be-406d-85fd-ef724cde8199 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:26.674564 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fea8e8b-b3be-406d-85fd-ef724cde8199 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:26.692722 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-6fea8e8b-b3be-406d-85fd-ef724cde8199 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:00:26.693409 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:26.693929 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-6fea8e8b-b3be-406d-85fd-ef724cde8199 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:00:26.761228 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 497/984] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:00:26 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 05:00:26.762067 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fea8e8b-b3be-406d-85fd-ef724cde8199 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:26.779661 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70a4e3f7-0192-4142-8c68-a2198e068154 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:26.815927 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-060dcf55-b4b8-4143-a10b-e1d3a943d6bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:26.816525 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-060dcf55-b4b8-4143-a10b-e1d3a943d6bc None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:26.816802 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-060dcf55-b4b8-4143-a10b-e1d3a943d6bc None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:26.817190 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-060dcf55-b4b8-4143-a10b-e1d3a943d6bc None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:26.817908 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-060dcf55-b4b8-4143-a10b-e1d3a943d6bc None admin] RBAC: Adding query filter params (role.id=2d3c2095a28d4f4f8a8328c4b61b0015, scope.project.id=e9919141e3b947ab8d91b55bba4a7de6, user.id=fe51dbec91624ed1ab00b441e29a3dd5) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:26.818213 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-060dcf55-b4b8-4143-a10b-e1d3a943d6bc None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:26.820341 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-060dcf55-b4b8-4143-a10b-e1d3a943d6bc None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:26.839033 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 488/985] 10.222.0.77 () {56 vars in 1374 bytes} [Tue Aug 30 05:00:26 2022] GET /identity/v3/role_assignments?user.id=fe51dbec91624ed1ab00b441e29a3dd5&scope.project.id=e9919141e3b947ab8d91b55bba4a7de6&role.id=2d3c2095a28d4f4f8a8328c4b61b0015 => generated 608 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:26.868999 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fea8e8b-b3be-406d-85fd-ef724cde8199 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:26.899786 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:26.909703 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91df0524-d56f-478f-b715-306fa58d9a1d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:26.910323 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91df0524-d56f-478f-b715-306fa58d9a1d None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:26.910609 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91df0524-d56f-478f-b715-306fa58d9a1d None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:26.910905 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91df0524-d56f-478f-b715-306fa58d9a1d None admin] PATH_INFO: `/v3/services` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:26.911544 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91df0524-d56f-478f-b715-306fa58d9a1d None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:26.911844 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91df0524-d56f-478f-b715-306fa58d9a1d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:26.913225 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91df0524-d56f-478f-b715-306fa58d9a1d None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:26.923903 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 498/986] 10.222.0.77 () {56 vars in 1122 bytes} [Tue Aug 30 05:00:26 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:26.924988 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-060dcf55-b4b8-4143-a10b-e1d3a943d6bc None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:27.017541 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:27.032696 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91df0524-d56f-478f-b715-306fa58d9a1d None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:27.037142 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-060dcf55-b4b8-4143-a10b-e1d3a943d6bc None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:27.073619 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76c52d8e-5243-4c51-83d8-8ecaff9ba709 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:27.074223 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c52d8e-5243-4c51-83d8-8ecaff9ba709 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:27.074501 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c52d8e-5243-4c51-83d8-8ecaff9ba709 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:27.074770 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c52d8e-5243-4c51-83d8-8ecaff9ba709 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:27.075496 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76c52d8e-5243-4c51-83d8-8ecaff9ba709 None admin] RBAC: Adding query filter params (interface=admin, service_id=f46ae468494244dfac7071ab1f2efd69, region_id=RegionOne) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:27.075801 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76c52d8e-5243-4c51-83d8-8ecaff9ba709 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:27.077178 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76c52d8e-5243-4c51-83d8-8ecaff9ba709 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:27.088563 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 489/987] 10.222.0.77 () {56 vars in 1249 bytes} [Tue Aug 30 05:00:26 2022] GET /identity/v3/endpoints?service_id=f46ae468494244dfac7071ab1f2efd69&interface=admin®ion_id=RegionOne => generated 193 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 05:00:27.089296 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76c52d8e-5243-4c51-83d8-8ecaff9ba709 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:27.145805 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91df0524-d56f-478f-b715-306fa58d9a1d None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:27.182043 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18249781-f050-49fd-b05a-f0302dfb2a87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:27.182643 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18249781-f050-49fd-b05a-f0302dfb2a87 None admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:27.182949 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18249781-f050-49fd-b05a-f0302dfb2a87 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:27.183213 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18249781-f050-49fd-b05a-f0302dfb2a87 None admin] PATH_INFO: `/v3/services` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:27.183861 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18249781-f050-49fd-b05a-f0302dfb2a87 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:27.185326 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18249781-f050-49fd-b05a-f0302dfb2a87 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:27.192916 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76c52d8e-5243-4c51-83d8-8ecaff9ba709 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:27.206861 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 499/988] 10.222.0.77 () {60 vars in 1141 bytes} [Tue Aug 30 05:00:27 2022] POST /identity/v3/services => generated 242 bytes in 171 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 05:00:27.241696 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0be9d73-7ead-419e-90f2-80f529a9993d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:27.242312 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0be9d73-7ead-419e-90f2-80f529a9993d None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:27.242589 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0be9d73-7ead-419e-90f2-80f529a9993d None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:27.242905 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0be9d73-7ead-419e-90f2-80f529a9993d None admin] PATH_INFO: `/v3/services` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:27.243523 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0be9d73-7ead-419e-90f2-80f529a9993d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:27.244975 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0be9d73-7ead-419e-90f2-80f529a9993d None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:27.258426 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 490/989] 10.222.0.77 () {56 vars in 1087 bytes} [Tue Aug 30 05:00:27 2022] GET /identity/v3/services => generated 1727 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:00:28.776524 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e17f0c1-dc66-4215-9bde-250191f28b99 None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:28.777015 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e17f0c1-dc66-4215-9bde-250191f28b99 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:28.777015 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e17f0c1-dc66-4215-9bde-250191f28b99 None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:28.778327 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 500/990] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:00:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:00:28.783321 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb995a4-c1ab-46a2-8631-4ad41a36d400 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:28.783700 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb995a4-c1ab-46a2-8631-4ad41a36d400 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:28.783860 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb995a4-c1ab-46a2-8631-4ad41a36d400 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:28.797025 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:28.803294 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-bbb995a4-c1ab-46a2-8631-4ad41a36d400 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:00:28.804844 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-bbb995a4-c1ab-46a2-8631-4ad41a36d400 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:00:28.873587 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 491/991] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:00:28 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 05:00:29.131105 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-552905ad-8a99-4207-850c-d7c6bdf304dd None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:29.132033 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-552905ad-8a99-4207-850c-d7c6bdf304dd None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:29.132238 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-552905ad-8a99-4207-850c-d7c6bdf304dd None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:29.133719 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 501/992] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:00:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:00:29.138791 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f863d5e1-7b56-41de-ae78-aae5c682ba40 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:29.139296 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f863d5e1-7b56-41de-ae78-aae5c682ba40 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:29.139467 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f863d5e1-7b56-41de-ae78-aae5c682ba40 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:29.140092 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-552905ad-8a99-4207-850c-d7c6bdf304dd None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:29.157326 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-f863d5e1-7b56-41de-ae78-aae5c682ba40 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:00:29.158613 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-f863d5e1-7b56-41de-ae78-aae5c682ba40 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:00:29.194476 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:29.225211 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 492/993] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:00:29 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 05:00:29.226488 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08b50f99-b0d4-4bdc-a489-2ac13c1dabce None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:29.226769 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08b50f99-b0d4-4bdc-a489-2ac13c1dabce None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:29.227081 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08b50f99-b0d4-4bdc-a489-2ac13c1dabce None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:29.231574 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 493/994] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:00:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:00:29.233827 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b7cf12d-5abf-471e-b785-454b29f57757 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:29.234110 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b7cf12d-5abf-471e-b785-454b29f57757 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:29.234382 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b7cf12d-5abf-471e-b785-454b29f57757 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:29.250467 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-552905ad-8a99-4207-850c-d7c6bdf304dd None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:29.253390 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-4b7cf12d-5abf-471e-b785-454b29f57757 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:00:29.254576 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-4b7cf12d-5abf-471e-b785-454b29f57757 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:00:29.289288 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7ac3dcd-f60f-4c2a-9153-75f19295f84c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:29.289908 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7ac3dcd-f60f-4c2a-9153-75f19295f84c None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:29.290213 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7ac3dcd-f60f-4c2a-9153-75f19295f84c None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:29.290503 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7ac3dcd-f60f-4c2a-9153-75f19295f84c None admin] PATH_INFO: `/v3/services/image` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:29.291033 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7ac3dcd-f60f-4c2a-9153-75f19295f84c None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:29.292384 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7ac3dcd-f60f-4c2a-9153-75f19295f84c None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:29.301866 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-a7ac3dcd-f60f-4c2a-9153-75f19295f84c None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Aug 30 05:00:29.301866 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:29.301866 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 30 05:00:29.301866 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 30 05:00:29.301866 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 30 05:00:29.301866 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 30 05:00:29.301866 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 30 05:00:29.301866 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 05:00:29.301866 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Aug 30 05:00:29.301866 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:00:29.301866 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 05:00:29.301866 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:00:29.301866 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:29.301866 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:00:29.301866 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:00:29.301866 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:00:29.301866 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:00:29.302911 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:00:29.302911 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:00:29.302911 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:00:29.302911 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:00:29.302911 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:00:29.302911 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:00:29.302911 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 30 05:00:29.302911 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 30 05:00:29.302911 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 30 05:00:29.302911 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 30 05:00:29.302911 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:00:29.302911 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:00:29.302911 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:00:29.302911 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:00:29.302911 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:00:29.302911 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:00:29.304365 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:00:29.304365 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 05:00:29.304365 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:00:29.304365 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:00:29.304365 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:00:29.304365 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:00:29.304365 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:00:29.304365 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:00:29.304365 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:00:29.304365 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:00:29.304365 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 30 05:00:29.304365 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 05:00:29.304365 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Aug 30 05:00:29.304365 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:00:29.304365 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 502/995] 10.222.0.77 () {56 vars in 1105 bytes} [Tue Aug 30 05:00:29 2022] GET /identity/v3/services/image => generated 86 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 05:00:29.309172 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7ac3dcd-f60f-4c2a-9153-75f19295f84c None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:29.323997 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:29.332733 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 494/996] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:00:29 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 30 05:00:29.414363 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7ac3dcd-f60f-4c2a-9153-75f19295f84c None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:29.431613 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:29.450070 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff79ce0b-15fd-4b17-ad73-b881fded3e0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:29.450669 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff79ce0b-15fd-4b17-ad73-b881fded3e0e None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:29.450978 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff79ce0b-15fd-4b17-ad73-b881fded3e0e None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:29.451249 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff79ce0b-15fd-4b17-ad73-b881fded3e0e None admin] PATH_INFO: `/v3/services` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:29.451867 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff79ce0b-15fd-4b17-ad73-b881fded3e0e None admin] RBAC: Adding query filter params (name=image) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:29.452167 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff79ce0b-15fd-4b17-ad73-b881fded3e0e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:29.454893 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff79ce0b-15fd-4b17-ad73-b881fded3e0e None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:29.462231 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 503/997] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 05:00:29 2022] GET /identity/v3/services?name=image => generated 122 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 05:00:29.467457 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b7cf12d-5abf-471e-b785-454b29f57757 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:29.497644 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff79ce0b-15fd-4b17-ad73-b881fded3e0e None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:29.572424 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b7cf12d-5abf-471e-b785-454b29f57757 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:29.599515 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff79ce0b-15fd-4b17-ad73-b881fded3e0e None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:29.608234 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5791396-bcc3-4adc-9b27-6daa6d8d64a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:29.608833 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5791396-bcc3-4adc-9b27-6daa6d8d64a6 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:29.609109 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5791396-bcc3-4adc-9b27-6daa6d8d64a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:29.609376 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5791396-bcc3-4adc-9b27-6daa6d8d64a6 None admin] PATH_INFO: `/v3/services` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:29.610106 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5791396-bcc3-4adc-9b27-6daa6d8d64a6 None admin] RBAC: Adding query filter params (type=image) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:29.610405 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5791396-bcc3-4adc-9b27-6daa6d8d64a6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:29.611898 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5791396-bcc3-4adc-9b27-6daa6d8d64a6 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:29.616871 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 495/998] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 05:00:29 2022] GET /identity/v3/services?type=image => generated 122 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:29.617641 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5791396-bcc3-4adc-9b27-6daa6d8d64a6 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:29.648577 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecf340d1-260e-4e22-b90a-55c2a1fa2bbe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:29.649221 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf340d1-260e-4e22-b90a-55c2a1fa2bbe None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:29.649506 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf340d1-260e-4e22-b90a-55c2a1fa2bbe None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:29.649777 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf340d1-260e-4e22-b90a-55c2a1fa2bbe None admin] PATH_INFO: `/v3/services/compute` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:29.650375 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecf340d1-260e-4e22-b90a-55c2a1fa2bbe None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:29.651831 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecf340d1-260e-4e22-b90a-55c2a1fa2bbe None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:29.661946 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-ecf340d1-260e-4e22-b90a-55c2a1fa2bbe None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Aug 30 05:00:29.661946 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:29.661946 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 30 05:00:29.661946 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 30 05:00:29.661946 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 30 05:00:29.661946 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 30 05:00:29.661946 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 30 05:00:29.661946 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 05:00:29.661946 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Aug 30 05:00:29.661946 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:00:29.661946 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 05:00:29.661946 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:00:29.661946 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:29.661946 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:00:29.661946 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:00:29.661946 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:00:29.661946 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:00:29.663639 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:00:29.663639 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:00:29.663639 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:00:29.663639 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:00:29.663639 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:00:29.663639 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:00:29.663639 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 30 05:00:29.663639 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 30 05:00:29.663639 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 30 05:00:29.663639 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 30 05:00:29.663639 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:00:29.663639 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:00:29.663639 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:00:29.663639 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:00:29.663639 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:00:29.663639 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:00:29.668789 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:00:29.668789 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 05:00:29.668789 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:00:29.668789 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:00:29.668789 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:00:29.668789 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:00:29.668789 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:00:29.668789 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:00:29.668789 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:00:29.668789 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:00:29.668789 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 30 05:00:29.668789 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 05:00:29.668789 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Aug 30 05:00:29.668789 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:00:29.668789 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 504/999] 10.222.0.77 () {56 vars in 1111 bytes} [Tue Aug 30 05:00:29 2022] GET /identity/v3/services/compute => generated 88 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 05:00:29.670163 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecf340d1-260e-4e22-b90a-55c2a1fa2bbe None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:29.690654 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:29.722508 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:29.741851 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5791396-bcc3-4adc-9b27-6daa6d8d64a6 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:29.785519 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab991737-4cb9-4839-b282-35b2c725af41 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:29.786257 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab991737-4cb9-4839-b282-35b2c725af41 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:29.786614 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab991737-4cb9-4839-b282-35b2c725af41 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:29.786925 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab991737-4cb9-4839-b282-35b2c725af41 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:29.787569 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab991737-4cb9-4839-b282-35b2c725af41 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:29.789445 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab991737-4cb9-4839-b282-35b2c725af41 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:29.791296 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecf340d1-260e-4e22-b90a-55c2a1fa2bbe None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:29.798238 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-ab991737-4cb9-4839-b282-35b2c725af41 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 30 05:00:29.798238 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:29.798238 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 30 05:00:29.798238 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 30 05:00:29.798238 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 30 05:00:29.798238 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 30 05:00:29.798238 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 30 05:00:29.798238 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 05:00:29.798238 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 30 05:00:29.798238 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:00:29.798238 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 05:00:29.798238 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:00:29.798238 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:29.798238 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:00:29.798238 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:00:29.798238 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:00:29.798238 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:00:29.799270 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:00:29.799270 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:00:29.799270 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:00:29.799270 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:00:29.799270 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:00:29.799270 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:00:29.799270 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 30 05:00:29.799270 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 30 05:00:29.799270 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 30 05:00:29.799270 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 30 05:00:29.799270 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:00:29.799270 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:00:29.799270 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:00:29.799270 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:00:29.799270 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:00:29.799270 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:00:29.800126 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:00:29.800126 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 05:00:29.800126 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:00:29.800126 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:00:29.800126 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:00:29.800126 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:00:29.800126 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:00:29.800126 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:00:29.800126 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:00:29.800126 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:00:29.800126 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 30 05:00:29.800126 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 05:00:29.800126 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 30 05:00:29.800126 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:00:29.804451 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 496/1000] 10.222.0.77 () {56 vars in 1126 bytes} [Tue Aug 30 05:00:29 2022] GET /identity/v3/services/object-store => generated 93 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:00:29.805125 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab991737-4cb9-4839-b282-35b2c725af41 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:29.830447 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86a73a40-1567-416e-b097-2ddc4757408f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:29.831086 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a73a40-1567-416e-b097-2ddc4757408f None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:29.831366 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a73a40-1567-416e-b097-2ddc4757408f None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:29.831642 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a73a40-1567-416e-b097-2ddc4757408f None admin] PATH_INFO: `/v3/services` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:29.832258 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86a73a40-1567-416e-b097-2ddc4757408f None admin] RBAC: Adding query filter params (name=compute) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:29.832557 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86a73a40-1567-416e-b097-2ddc4757408f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:29.833918 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86a73a40-1567-416e-b097-2ddc4757408f None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:29.839213 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 505/1001] 10.222.0.77 () {56 vars in 1112 bytes} [Tue Aug 30 05:00:29 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:29.844613 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86a73a40-1567-416e-b097-2ddc4757408f None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:29.921095 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab991737-4cb9-4839-b282-35b2c725af41 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:29.947605 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:29.951268 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86a73a40-1567-416e-b097-2ddc4757408f None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:29.960376 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da77edaa-ee37-419b-b0b4-83b91d5d638a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:29.960973 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da77edaa-ee37-419b-b0b4-83b91d5d638a None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:29.961246 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da77edaa-ee37-419b-b0b4-83b91d5d638a None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:29.961510 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da77edaa-ee37-419b-b0b4-83b91d5d638a None admin] PATH_INFO: `/v3/services` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:29.962130 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da77edaa-ee37-419b-b0b4-83b91d5d638a None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:29.962421 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da77edaa-ee37-419b-b0b4-83b91d5d638a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:29.962521 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:29.963892 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da77edaa-ee37-419b-b0b4-83b91d5d638a None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:29.973864 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 497/1002] 10.222.0.77 () {56 vars in 1122 bytes} [Tue Aug 30 05:00:29 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:29.974806 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da77edaa-ee37-419b-b0b4-83b91d5d638a None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:29.995586 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c77104f5-9a28-49ba-b397-a3f194ab13f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:29.996188 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c77104f5-9a28-49ba-b397-a3f194ab13f3 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:29.996467 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c77104f5-9a28-49ba-b397-a3f194ab13f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:29.996734 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c77104f5-9a28-49ba-b397-a3f194ab13f3 None admin] PATH_INFO: `/v3/services` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:29.997350 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c77104f5-9a28-49ba-b397-a3f194ab13f3 None admin] RBAC: Adding query filter params (type=compute) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:29.997645 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c77104f5-9a28-49ba-b397-a3f194ab13f3 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:29.999726 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c77104f5-9a28-49ba-b397-a3f194ab13f3 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:30.006971 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 506/1003] 10.222.0.77 () {56 vars in 1112 bytes} [Tue Aug 30 05:00:29 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 05:00:30.012351 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c77104f5-9a28-49ba-b397-a3f194ab13f3 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:30.086800 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da77edaa-ee37-419b-b0b4-83b91d5d638a None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:30.121462 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c77104f5-9a28-49ba-b397-a3f194ab13f3 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:30.123268 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b8e49c0-0475-4508-90bc-8125d724954e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:30.123876 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b8e49c0-0475-4508-90bc-8125d724954e None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:30.124149 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b8e49c0-0475-4508-90bc-8125d724954e None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:30.124415 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b8e49c0-0475-4508-90bc-8125d724954e None admin] PATH_INFO: `/v3/services` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:30.125036 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b8e49c0-0475-4508-90bc-8125d724954e None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:30.125339 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b8e49c0-0475-4508-90bc-8125d724954e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:30.126695 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b8e49c0-0475-4508-90bc-8125d724954e None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:30.132139 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 498/1004] 10.222.0.77 () {56 vars in 1122 bytes} [Tue Aug 30 05:00:29 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 05:00:30.137662 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b8e49c0-0475-4508-90bc-8125d724954e None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:30.159730 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae25ec54-cfe6-4e6e-ac2e-4af756eb1d8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:30.160333 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae25ec54-cfe6-4e6e-ac2e-4af756eb1d8e None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:30.160608 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae25ec54-cfe6-4e6e-ac2e-4af756eb1d8e None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:30.161206 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae25ec54-cfe6-4e6e-ac2e-4af756eb1d8e None admin] PATH_INFO: `/v3/endpoints` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:30.161546 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae25ec54-cfe6-4e6e-ac2e-4af756eb1d8e None admin] RBAC: Adding query filter params (interface=public, service_id=5a460d0eeb8c44ba9efeece11f773062, region_id=RegionOne) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:30.161845 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae25ec54-cfe6-4e6e-ac2e-4af756eb1d8e None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:30.163251 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae25ec54-cfe6-4e6e-ac2e-4af756eb1d8e None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:30.168955 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 507/1005] 10.222.0.77 () {56 vars in 1251 bytes} [Tue Aug 30 05:00:30 2022] GET /identity/v3/endpoints?service_id=5a460d0eeb8c44ba9efeece11f773062&interface=public®ion_id=RegionOne => generated 194 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:30.174156 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae25ec54-cfe6-4e6e-ac2e-4af756eb1d8e None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:30.192923 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:30.216575 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:30.245058 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b8e49c0-0475-4508-90bc-8125d724954e None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:30.282885 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ee2e2ee-2f69-4bca-9f27-8d155e3a642c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:30.283790 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ee2e2ee-2f69-4bca-9f27-8d155e3a642c None admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:30.284279 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ee2e2ee-2f69-4bca-9f27-8d155e3a642c None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:30.284737 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ee2e2ee-2f69-4bca-9f27-8d155e3a642c None admin] PATH_INFO: `/v3/endpoints` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:30.285154 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae25ec54-cfe6-4e6e-ac2e-4af756eb1d8e None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:30.285789 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ee2e2ee-2f69-4bca-9f27-8d155e3a642c None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:30.287394 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ee2e2ee-2f69-4bca-9f27-8d155e3a642c None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:30.314858 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 499/1006] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:00:30 2022] POST /identity/v3/endpoints => generated 325 bytes in 167 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:00:30.339911 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ab946f2-cd73-4a78-981a-adf958a72f08 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:30.340635 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab946f2-cd73-4a78-981a-adf958a72f08 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:30.340987 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab946f2-cd73-4a78-981a-adf958a72f08 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:30.341336 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab946f2-cd73-4a78-981a-adf958a72f08 None admin] PATH_INFO: `/v3/services` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:30.342025 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ab946f2-cd73-4a78-981a-adf958a72f08 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:30.343527 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ab946f2-cd73-4a78-981a-adf958a72f08 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:30.354852 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 508/1007] 10.222.0.77 () {56 vars in 1087 bytes} [Tue Aug 30 05:00:30 2022] GET /identity/v3/services => generated 1727 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:00:31.516543 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9976c66-2bfc-4bdc-b437-b367f7f8fc9f None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:31.517044 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9976c66-2bfc-4bdc-b437-b367f7f8fc9f None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:31.517120 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9976c66-2bfc-4bdc-b437-b367f7f8fc9f None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:31.518591 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 500/1008] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:00:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 05:00:31.523433 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86b14f1a-03b0-4cc2-9969-c6937e747629 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:31.523801 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86b14f1a-03b0-4cc2-9969-c6937e747629 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:31.523985 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86b14f1a-03b0-4cc2-9969-c6937e747629 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:31.542577 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-86b14f1a-03b0-4cc2-9969-c6937e747629 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:00:31.543785 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-86b14f1a-03b0-4cc2-9969-c6937e747629 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:00:31.609005 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 509/1009] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:00:31 2022] POST /identity/v3/auth/tokens => generated 1717 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 05:00:31.872465 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9976c66-2bfc-4bdc-b437-b367f7f8fc9f None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:31.975676 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9976c66-2bfc-4bdc-b437-b367f7f8fc9f None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:31.988807 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:32.010585 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2d0a0a9-c792-407a-b5cb-0ca528b9c1a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:32.011213 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d0a0a9-c792-407a-b5cb-0ca528b9c1a0 None admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:32.011492 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d0a0a9-c792-407a-b5cb-0ca528b9c1a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:32.011758 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d0a0a9-c792-407a-b5cb-0ca528b9c1a0 None admin] PATH_INFO: `/v3/services` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:32.012398 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2d0a0a9-c792-407a-b5cb-0ca528b9c1a0 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:32.013759 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2d0a0a9-c792-407a-b5cb-0ca528b9c1a0 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:32.030860 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 501/1010] 10.222.0.77 () {60 vars in 1141 bytes} [Tue Aug 30 05:00:31 2022] POST /identity/v3/services => generated 242 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:00:32.231200 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ee4517-c3bf-4eb6-9ffc-8412087d4ae4 None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:32.231389 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ee4517-c3bf-4eb6-9ffc-8412087d4ae4 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:32.231655 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ee4517-c3bf-4eb6-9ffc-8412087d4ae4 None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:32.233145 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 510/1011] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:00:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:00:32.238124 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1030fc9-708b-49bd-b2ca-e4e8f5557b64 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:32.238417 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1030fc9-708b-49bd-b2ca-e4e8f5557b64 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:32.238695 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1030fc9-708b-49bd-b2ca-e4e8f5557b64 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:32.256733 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-f1030fc9-708b-49bd-b2ca-e4e8f5557b64 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:00:32.257930 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-f1030fc9-708b-49bd-b2ca-e4e8f5557b64 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:00:32.284227 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f05fca7-e367-493a-81d5-25f0387a9dd5 None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:32.284511 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f05fca7-e367-493a-81d5-25f0387a9dd5 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:32.284796 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f05fca7-e367-493a-81d5-25f0387a9dd5 None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:32.286099 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 511/1012] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:00:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 05:00:32.292104 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9a44565-9215-415a-979e-f4b835e575e3 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:32.292390 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9a44565-9215-415a-979e-f4b835e575e3 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:32.292790 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9a44565-9215-415a-979e-f4b835e575e3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:32.311350 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-b9a44565-9215-415a-979e-f4b835e575e3 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:00:32.313007 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-b9a44565-9215-415a-979e-f4b835e575e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:00:32.336098 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 502/1013] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:00:32 2022] POST /identity/v3/auth/tokens => generated 1813 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 05:00:32.340193 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:32.390857 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 512/1014] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:00:32 2022] POST /identity/v3/auth/tokens => generated 1813 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 30 05:00:32.607179 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1030fc9-708b-49bd-b2ca-e4e8f5557b64 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:32.660982 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9a44565-9215-415a-979e-f4b835e575e3 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:32.711928 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:32.716977 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1030fc9-708b-49bd-b2ca-e4e8f5557b64 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:32.753120 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-557599d8-39fa-4c7c-a6ad-bcbdfd470c5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:32.753750 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-557599d8-39fa-4c7c-a6ad-bcbdfd470c5a None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:32.754033 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-557599d8-39fa-4c7c-a6ad-bcbdfd470c5a None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:32.754301 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-557599d8-39fa-4c7c-a6ad-bcbdfd470c5a None admin] PATH_INFO: `/v3/domains/default` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:32.755445 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-557599d8-39fa-4c7c-a6ad-bcbdfd470c5a None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:32.757467 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-557599d8-39fa-4c7c-a6ad-bcbdfd470c5a None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:32.759191 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 503/1015] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 05:00:32 2022] GET /identity/v3/domains/default => generated 199 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:32.764879 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-557599d8-39fa-4c7c-a6ad-bcbdfd470c5a None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:32.771748 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9a44565-9215-415a-979e-f4b835e575e3 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:32.808978 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6053e2c-68c4-45a9-a315-e07370cbb268 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:32.809846 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6053e2c-68c4-45a9-a315-e07370cbb268 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:32.810382 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6053e2c-68c4-45a9-a315-e07370cbb268 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:32.811011 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6053e2c-68c4-45a9-a315-e07370cbb268 None admin] PATH_INFO: `/v3/services/compute` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:32.811592 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6053e2c-68c4-45a9-a315-e07370cbb268 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:32.812959 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6053e2c-68c4-45a9-a315-e07370cbb268 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:32.822999 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-d6053e2c-68c4-45a9-a315-e07370cbb268 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Aug 30 05:00:32.822999 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:32.822999 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 30 05:00:32.822999 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 30 05:00:32.822999 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 30 05:00:32.822999 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 30 05:00:32.822999 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 30 05:00:32.822999 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 05:00:32.822999 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Aug 30 05:00:32.822999 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:00:32.822999 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 05:00:32.822999 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:00:32.822999 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:32.822999 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:00:32.822999 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:00:32.822999 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:00:32.822999 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:00:32.824026 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:00:32.824026 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:00:32.824026 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:00:32.824026 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:00:32.824026 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:00:32.824026 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:00:32.824026 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 30 05:00:32.824026 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 30 05:00:32.824026 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 30 05:00:32.824026 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 30 05:00:32.824026 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:00:32.824026 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:00:32.824026 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:00:32.824026 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:00:32.824026 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:00:32.824026 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:00:32.824872 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:00:32.824872 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 05:00:32.824872 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:00:32.824872 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:00:32.824872 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:00:32.824872 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:00:32.824872 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:00:32.824872 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:00:32.824872 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:00:32.824872 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:00:32.824872 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 30 05:00:32.824872 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 05:00:32.824872 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Aug 30 05:00:32.824872 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:00:32.824872 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 513/1016] 10.222.0.77 () {56 vars in 1111 bytes} [Tue Aug 30 05:00:32 2022] GET /identity/v3/services/compute => generated 88 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 05:00:32.830453 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6053e2c-68c4-45a9-a315-e07370cbb268 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:32.853448 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:32.870021 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-557599d8-39fa-4c7c-a6ad-bcbdfd470c5a None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:32.907430 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec5f9970-047d-4d1e-a209-9581e1c88c81 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:32.908155 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec5f9970-047d-4d1e-a209-9581e1c88c81 None admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:32.908439 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec5f9970-047d-4d1e-a209-9581e1c88c81 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:32.908708 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec5f9970-047d-4d1e-a209-9581e1c88c81 None admin] PATH_INFO: `/v3/projects` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:32.909353 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec5f9970-047d-4d1e-a209-9581e1c88c81 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:32.910939 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec5f9970-047d-4d1e-a209-9581e1c88c81 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:32.934790 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6053e2c-68c4-45a9-a315-e07370cbb268 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:32.950852 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 504/1017] 10.222.0.77 () {60 vars in 1141 bytes} [Tue Aug 30 05:00:32 2022] POST /identity/v3/projects => generated 311 bytes in 176 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 05:00:32.983099 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-666fc931-39fe-4b52-a421-1a85231e4890 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:32.983833 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-666fc931-39fe-4b52-a421-1a85231e4890 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:32.984187 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-666fc931-39fe-4b52-a421-1a85231e4890 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:32.984534 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-666fc931-39fe-4b52-a421-1a85231e4890 None admin] PATH_INFO: `/v3/services` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:32.985260 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-666fc931-39fe-4b52-a421-1a85231e4890 None admin] RBAC: Adding query filter params (name=compute) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:32.985647 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-666fc931-39fe-4b52-a421-1a85231e4890 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:32.987234 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-666fc931-39fe-4b52-a421-1a85231e4890 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:32.998810 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 514/1018] 10.222.0.77 () {56 vars in 1112 bytes} [Tue Aug 30 05:00:32 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:33.000123 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec5f9970-047d-4d1e-a209-9581e1c88c81 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:33.022404 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:33.118546 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec5f9970-047d-4d1e-a209-9581e1c88c81 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:33.154572 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2beb664a-5f1e-46b0-b605-767d7480cd15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:33.155133 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2beb664a-5f1e-46b0-b605-767d7480cd15 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:33.155412 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2beb664a-5f1e-46b0-b605-767d7480cd15 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:33.155684 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2beb664a-5f1e-46b0-b605-767d7480cd15 None admin] PATH_INFO: `/v3/services` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:33.156310 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2beb664a-5f1e-46b0-b605-767d7480cd15 None admin] RBAC: Adding query filter params (type=compute) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:33.156608 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2beb664a-5f1e-46b0-b605-767d7480cd15 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:33.157960 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2beb664a-5f1e-46b0-b605-767d7480cd15 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:33.163803 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 505/1019] 10.222.0.77 () {56 vars in 1112 bytes} [Tue Aug 30 05:00:32 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:33.169304 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-666fc931-39fe-4b52-a421-1a85231e4890 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:33.254697 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:33.271781 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-666fc931-39fe-4b52-a421-1a85231e4890 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:33.307726 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a74e537a-e28e-45aa-bf09-4dcfd2df4c71 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:33.308334 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a74e537a-e28e-45aa-bf09-4dcfd2df4c71 None admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:33.308611 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a74e537a-e28e-45aa-bf09-4dcfd2df4c71 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:33.308881 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a74e537a-e28e-45aa-bf09-4dcfd2df4c71 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:33.309531 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a74e537a-e28e-45aa-bf09-4dcfd2df4c71 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:33.310903 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a74e537a-e28e-45aa-bf09-4dcfd2df4c71 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:33.338848 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 515/1020] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:00:33 2022] POST /identity/v3/endpoints => generated 334 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 05:00:33.945353 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e766c9a-0622-4fc5-805d-b7881ba1be3a None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:33.945899 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e766c9a-0622-4fc5-805d-b7881ba1be3a None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:33.945899 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e766c9a-0622-4fc5-805d-b7881ba1be3a None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:33.947310 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 506/1021] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:00:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:00:33.952306 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a6f3cf4-c5a3-4ef5-a389-e452b11d3e9e None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:33.952703 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a6f3cf4-c5a3-4ef5-a389-e452b11d3e9e None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:33.952863 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a6f3cf4-c5a3-4ef5-a389-e452b11d3e9e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:33.971789 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-7a6f3cf4-c5a3-4ef5-a389-e452b11d3e9e None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:00:33.972968 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-7a6f3cf4-c5a3-4ef5-a389-e452b11d3e9e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:00:34.052457 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 516/1022] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:00:33 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 05:00:34.315521 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e766c9a-0622-4fc5-805d-b7881ba1be3a None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:34.402452 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:34.423011 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e766c9a-0622-4fc5-805d-b7881ba1be3a None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:34.465119 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3233cf6d-00e4-44c6-b431-78e21bcdbadb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:34.465759 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3233cf6d-00e4-44c6-b431-78e21bcdbadb None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:34.466039 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3233cf6d-00e4-44c6-b431-78e21bcdbadb None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:34.466310 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3233cf6d-00e4-44c6-b431-78e21bcdbadb None admin] PATH_INFO: `/v3/services/image` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:34.467283 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3233cf6d-00e4-44c6-b431-78e21bcdbadb None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:34.468275 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3233cf6d-00e4-44c6-b431-78e21bcdbadb None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:34.477628 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-3233cf6d-00e4-44c6-b431-78e21bcdbadb None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Aug 30 05:00:34.477628 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:34.477628 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 30 05:00:34.477628 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 30 05:00:34.477628 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 30 05:00:34.477628 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 30 05:00:34.477628 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 30 05:00:34.477628 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 05:00:34.477628 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Aug 30 05:00:34.477628 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:00:34.477628 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 05:00:34.477628 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:00:34.477628 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:34.477628 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:00:34.477628 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:00:34.477628 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:00:34.477628 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:00:34.483112 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:00:34.483112 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:00:34.483112 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:00:34.483112 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:00:34.483112 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:00:34.483112 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:00:34.483112 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 30 05:00:34.483112 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 30 05:00:34.483112 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 30 05:00:34.483112 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 30 05:00:34.483112 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:00:34.483112 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:00:34.483112 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:00:34.483112 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:00:34.483112 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:00:34.483112 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:00:34.484024 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:00:34.484024 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 05:00:34.484024 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:00:34.484024 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:00:34.484024 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:00:34.484024 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:00:34.484024 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:00:34.484024 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:00:34.484024 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:00:34.484024 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:00:34.484024 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 30 05:00:34.484024 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 05:00:34.484024 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Aug 30 05:00:34.484024 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:00:34.484024 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 507/1023] 10.222.0.77 () {56 vars in 1105 bytes} [Tue Aug 30 05:00:34 2022] GET /identity/v3/services/image => generated 86 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:00:34.485274 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a6f3cf4-c5a3-4ef5-a389-e452b11d3e9e None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:34.553620 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:34.589514 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a6f3cf4-c5a3-4ef5-a389-e452b11d3e9e None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:34.624458 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01332e59-305a-42d9-a64c-a31fe85be4b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:34.625060 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01332e59-305a-42d9-a64c-a31fe85be4b6 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:34.625335 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01332e59-305a-42d9-a64c-a31fe85be4b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:34.625601 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01332e59-305a-42d9-a64c-a31fe85be4b6 None admin] PATH_INFO: `/v3/services` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:34.626213 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01332e59-305a-42d9-a64c-a31fe85be4b6 None admin] RBAC: Adding query filter params (name=image) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:34.626505 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01332e59-305a-42d9-a64c-a31fe85be4b6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:34.627889 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01332e59-305a-42d9-a64c-a31fe85be4b6 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:34.637245 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 517/1024] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 05:00:34 2022] GET /identity/v3/services?name=image => generated 122 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:34.638439 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3233cf6d-00e4-44c6-b431-78e21bcdbadb None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:34.740144 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3233cf6d-00e4-44c6-b431-78e21bcdbadb None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:34.775916 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efe0514f-a5bf-4dac-8357-7afdff20859e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:34.776519 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efe0514f-a5bf-4dac-8357-7afdff20859e None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:34.776790 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efe0514f-a5bf-4dac-8357-7afdff20859e None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:34.777109 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efe0514f-a5bf-4dac-8357-7afdff20859e None admin] PATH_INFO: `/v3/services` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:34.777729 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efe0514f-a5bf-4dac-8357-7afdff20859e None admin] RBAC: Adding query filter params (type=image) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:34.778075 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efe0514f-a5bf-4dac-8357-7afdff20859e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:34.779529 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efe0514f-a5bf-4dac-8357-7afdff20859e None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:34.790031 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 508/1025] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 05:00:34 2022] GET /identity/v3/services?type=image => generated 350 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:34.791186 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01332e59-305a-42d9-a64c-a31fe85be4b6 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:34.892884 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01332e59-305a-42d9-a64c-a31fe85be4b6 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:34.923451 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:34.929157 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f1ab924-9f27-44ca-9e93-aa2e98f92983 None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:34.929457 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f1ab924-9f27-44ca-9e93-aa2e98f92983 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:34.929706 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f1ab924-9f27-44ca-9e93-aa2e98f92983 None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:34.930630 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62a04bb1-f404-45c6-864c-4b36a480ea7a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:34.931099 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 509/1026] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:00:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 05:00:34.931265 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a04bb1-f404-45c6-864c-4b36a480ea7a None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:34.931534 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a04bb1-f404-45c6-864c-4b36a480ea7a None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:34.934848 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a04bb1-f404-45c6-864c-4b36a480ea7a None admin] PATH_INFO: `/v3/endpoints` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:34.934848 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62a04bb1-f404-45c6-864c-4b36a480ea7a None admin] RBAC: Adding query filter params (interface=public, service_id=615516ef3229445eb6425220997f6a5b, region_id=RegionOne) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:34.934848 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62a04bb1-f404-45c6-864c-4b36a480ea7a None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:34.934848 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62a04bb1-f404-45c6-864c-4b36a480ea7a None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:34.936575 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63cfffdf-7699-4940-87f9-f05ff8f6506f None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:34.936865 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63cfffdf-7699-4940-87f9-f05ff8f6506f None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:34.937136 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63cfffdf-7699-4940-87f9-f05ff8f6506f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:34.944592 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 518/1027] 10.222.0.77 () {56 vars in 1251 bytes} [Tue Aug 30 05:00:34 2022] GET /identity/v3/endpoints?service_id=615516ef3229445eb6425220997f6a5b&interface=public®ion_id=RegionOne => generated 194 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:34.945558 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62a04bb1-f404-45c6-864c-4b36a480ea7a None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:34.959685 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-63cfffdf-7699-4940-87f9-f05ff8f6506f None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:00:34.960988 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-63cfffdf-7699-4940-87f9-f05ff8f6506f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:00:34.964007 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:35.039163 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 510/1028] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:00:34 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 30 05:00:35.049931 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62a04bb1-f404-45c6-864c-4b36a480ea7a None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:35.082908 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f8b91c6-c0d1-49f1-97ac-ee72498f98a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:35.083503 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f8b91c6-c0d1-49f1-97ac-ee72498f98a7 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:35.083769 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f8b91c6-c0d1-49f1-97ac-ee72498f98a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:35.084039 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f8b91c6-c0d1-49f1-97ac-ee72498f98a7 None admin] PATH_INFO: `/v3/services` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:35.085153 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f8b91c6-c0d1-49f1-97ac-ee72498f98a7 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:35.086500 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f8b91c6-c0d1-49f1-97ac-ee72498f98a7 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:35.099050 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 519/1029] 10.222.0.77 () {56 vars in 1087 bytes} [Tue Aug 30 05:00:34 2022] GET /identity/v3/services => generated 1957 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 05:00:35.297409 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63cfffdf-7699-4940-87f9-f05ff8f6506f None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:35.344458 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e304ac6-537b-4c21-9c23-3ccc619bde44 None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:35.344771 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e304ac6-537b-4c21-9c23-3ccc619bde44 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:35.345026 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e304ac6-537b-4c21-9c23-3ccc619bde44 None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:35.346364 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 520/1030] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:00:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 05:00:35.351312 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc58d4e0-c2e0-433a-8328-acacfd022fb7 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:35.351613 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc58d4e0-c2e0-433a-8328-acacfd022fb7 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:35.351876 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc58d4e0-c2e0-433a-8328-acacfd022fb7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:35.370439 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-bc58d4e0-c2e0-433a-8328-acacfd022fb7 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:00:35.371664 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-bc58d4e0-c2e0-433a-8328-acacfd022fb7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:00:35.400524 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63cfffdf-7699-4940-87f9-f05ff8f6506f None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:35.429104 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:35.444627 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ee70cf8-6026-4049-ac12-ef5345a55de2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:35.445243 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 521/1031] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:00:35 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 30 05:00:35.445376 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee70cf8-6026-4049-ac12-ef5345a55de2 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:35.445600 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee70cf8-6026-4049-ac12-ef5345a55de2 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:35.445873 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee70cf8-6026-4049-ac12-ef5345a55de2 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:35.447095 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ee70cf8-6026-4049-ac12-ef5345a55de2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:35.448532 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ee70cf8-6026-4049-ac12-ef5345a55de2 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:35.454806 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 511/1032] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 05:00:35 2022] GET /identity/v3/domains/default => generated 199 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:35.455969 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc58d4e0-c2e0-433a-8328-acacfd022fb7 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:35.558391 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc58d4e0-c2e0-433a-8328-acacfd022fb7 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:35.592464 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb8b8c64-934f-4855-8f9c-22e93dedc0ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:35.593058 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb8b8c64-934f-4855-8f9c-22e93dedc0ea None admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:35.593331 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb8b8c64-934f-4855-8f9c-22e93dedc0ea None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:35.593596 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb8b8c64-934f-4855-8f9c-22e93dedc0ea None admin] PATH_INFO: `/v3/users` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:35.594225 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb8b8c64-934f-4855-8f9c-22e93dedc0ea None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:35.595666 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb8b8c64-934f-4855-8f9c-22e93dedc0ea None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:35.618892 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 522/1033] 10.222.0.77 () {60 vars in 1132 bytes} [Tue Aug 30 05:00:35 2022] POST /identity/v3/users => generated 285 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:00:35.716533 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ee70cf8-6026-4049-ac12-ef5345a55de2 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:35.728299 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:35.822116 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ee70cf8-6026-4049-ac12-ef5345a55de2 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:35.855942 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40426eac-1311-4605-afa3-ec55a23bef41 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:35.856568 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40426eac-1311-4605-afa3-ec55a23bef41 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:35.856840 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40426eac-1311-4605-afa3-ec55a23bef41 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:35.857104 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40426eac-1311-4605-afa3-ec55a23bef41 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:35.867537 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40426eac-1311-4605-afa3-ec55a23bef41 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 05:00:35.872250 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-40426eac-1311-4605-afa3-ec55a23bef41 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 30 05:00:35.872250 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:35.872250 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:00:35.872250 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:00:35.872250 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:00:35.872250 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:00:35.872250 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:00:35.872250 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:00:35.872250 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:00:35.872250 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:00:35.872250 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:00:35.872250 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:00:35.872250 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 05:00:35.872250 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 05:00:35.872250 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 05:00:35.872250 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 05:00:35.873154 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 05:00:35.873154 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 05:00:35.873154 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 05:00:35.873154 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 05:00:35.873154 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:00:35.873154 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:00:35.873154 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:00:35.873154 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:00:35.873154 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:00:35.873154 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:00:35.873154 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:00:35.873154 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 05:00:35.873154 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:00:35.873154 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:00:35.873154 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:00:35.873154 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:00:35.873983 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:00:35.873983 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:00:35.873983 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:00:35.873983 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:00:35.873983 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 05:00:35.873983 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 05:00:35.873983 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 05:00:35.873983 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 05:00:35.873983 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 05:00:35.873983 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 05:00:35.873983 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 30 05:00:35.873983 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:00:35.873983 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 512/1034] 10.222.0.77 () {56 vars in 1120 bytes} [Tue Aug 30 05:00:35 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:00:35.879514 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb8b8c64-934f-4855-8f9c-22e93dedc0ea None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:35.927507 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:35.984628 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb8b8c64-934f-4855-8f9c-22e93dedc0ea None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:36.018129 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-600259b6-7b44-4dc3-8826-9a70e51726d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:36.018731 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-600259b6-7b44-4dc3-8826-9a70e51726d9 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:36.019037 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-600259b6-7b44-4dc3-8826-9a70e51726d9 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:36.019307 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-600259b6-7b44-4dc3-8826-9a70e51726d9 None admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:36.019959 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-600259b6-7b44-4dc3-8826-9a70e51726d9 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:36.020256 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-600259b6-7b44-4dc3-8826-9a70e51726d9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:36.021631 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-600259b6-7b44-4dc3-8826-9a70e51726d9 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:36.029943 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 523/1035] 10.222.0.77 () {56 vars in 1115 bytes} [Tue Aug 30 05:00:35 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 05:00:36.035780 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40426eac-1311-4605-afa3-ec55a23bef41 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:36.103670 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:36.142872 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40426eac-1311-4605-afa3-ec55a23bef41 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:36.179351 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beca29a6-b91e-4eda-b9c1-f35bc1ef74c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:36.179980 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beca29a6-b91e-4eda-b9c1-f35bc1ef74c6 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:36.180267 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beca29a6-b91e-4eda-b9c1-f35bc1ef74c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:36.180539 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beca29a6-b91e-4eda-b9c1-f35bc1ef74c6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:36.190653 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beca29a6-b91e-4eda-b9c1-f35bc1ef74c6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:36.192198 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beca29a6-b91e-4eda-b9c1-f35bc1ef74c6 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:36.206651 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-beca29a6-b91e-4eda-b9c1-f35bc1ef74c6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 05:00:36.206651 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:36.206651 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:00:36.206651 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:00:36.206651 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:00:36.206651 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:00:36.206651 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:00:36.206651 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:00:36.206651 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:00:36.206651 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:00:36.206651 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:00:36.206651 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:00:36.206651 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 05:00:36.206651 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 05:00:36.206651 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 05:00:36.206651 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 05:00:36.207630 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:00:36.207630 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:00:36.207630 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:00:36.207630 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:00:36.207630 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:00:36.207630 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:00:36.207630 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:00:36.207630 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 05:00:36.207630 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:00:36.207630 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:00:36.207630 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:00:36.207630 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:00:36.207630 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:00:36.207630 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:00:36.207630 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:00:36.207630 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:00:36.207630 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 05:00:36.208524 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 05:00:36.208524 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 05:00:36.208524 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:00:36.208524 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 513/1036] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 05:00:36 2022] GET /identity/v3/domains/Default => generated 87 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 05:00:36.214619 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-600259b6-7b44-4dc3-8826-9a70e51726d9 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:36.315584 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-600259b6-7b44-4dc3-8826-9a70e51726d9 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:36.322442 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:36.350750 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-977a6245-a82f-4657-b5d4-86bc19334890 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:36.351382 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-977a6245-a82f-4657-b5d4-86bc19334890 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:36.351656 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-977a6245-a82f-4657-b5d4-86bc19334890 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:36.351924 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-977a6245-a82f-4657-b5d4-86bc19334890 None admin] PATH_INFO: `/v3/domains` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:36.352550 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-977a6245-a82f-4657-b5d4-86bc19334890 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:36.353113 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-977a6245-a82f-4657-b5d4-86bc19334890 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:36.354520 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-977a6245-a82f-4657-b5d4-86bc19334890 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:36.366855 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 524/1037] 10.222.0.77 () {56 vars in 1109 bytes} [Tue Aug 30 05:00:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 05:00:36.372457 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beca29a6-b91e-4eda-b9c1-f35bc1ef74c6 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:36.478473 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beca29a6-b91e-4eda-b9c1-f35bc1ef74c6 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:36.498908 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:36.514131 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff201734-be16-4f6b-8541-e6dfa6fefcb3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:36.514883 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff201734-be16-4f6b-8541-e6dfa6fefcb3 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:36.515168 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff201734-be16-4f6b-8541-e6dfa6fefcb3 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:36.515438 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff201734-be16-4f6b-8541-e6dfa6fefcb3 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:36.523820 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff201734-be16-4f6b-8541-e6dfa6fefcb3 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:36.525268 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff201734-be16-4f6b-8541-e6dfa6fefcb3 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:36.536456 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-ff201734-be16-4f6b-8541-e6dfa6fefcb3 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 30 05:00:36.536456 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:36.536456 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:00:36.536456 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:00:36.536456 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:00:36.536456 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:00:36.536456 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:00:36.536456 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:00:36.536456 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:00:36.536456 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:00:36.536456 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:00:36.536456 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:00:36.536456 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 05:00:36.536456 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 05:00:36.536456 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 05:00:36.536456 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 05:00:36.538061 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:00:36.538061 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:00:36.538061 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 05:00:36.538061 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 05:00:36.538061 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 05:00:36.538061 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 05:00:36.538061 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:00:36.538061 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:00:36.538061 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:00:36.538061 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:00:36.538061 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:00:36.538061 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 05:00:36.538061 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:00:36.538061 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:00:36.538061 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:00:36.538061 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:00:36.538061 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:00:36.538061 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:00:36.543759 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:00:36.543759 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:00:36.543759 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 05:00:36.543759 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 05:00:36.543759 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 05:00:36.543759 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 05:00:36.543759 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 05:00:36.543759 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 05:00:36.543759 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 30 05:00:36.543759 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:00:36.543759 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 514/1038] 10.222.0.77 () {56 vars in 1093 bytes} [Tue Aug 30 05:00:36 2022] GET /identity/v3/users/nova => generated 82 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:00:36.546161 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-977a6245-a82f-4657-b5d4-86bc19334890 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:36.649338 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-977a6245-a82f-4657-b5d4-86bc19334890 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:36.682974 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46026424-0866-4cca-92d7-78749fc2f55f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:36.683574 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46026424-0866-4cca-92d7-78749fc2f55f None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:36.683846 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46026424-0866-4cca-92d7-78749fc2f55f None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:36.684107 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46026424-0866-4cca-92d7-78749fc2f55f None admin] PATH_INFO: `/v3/users` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:36.684822 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46026424-0866-4cca-92d7-78749fc2f55f None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:36.685116 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46026424-0866-4cca-92d7-78749fc2f55f None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:36.686518 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46026424-0866-4cca-92d7-78749fc2f55f None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:36.708374 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 525/1039] 10.222.0.77 () {56 vars in 1133 bytes} [Tue Aug 30 05:00:36 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:36.710143 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff201734-be16-4f6b-8541-e6dfa6fefcb3 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:36.816392 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff201734-be16-4f6b-8541-e6dfa6fefcb3 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:36.849860 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb0b0b50-a999-4b3a-a54f-7771247b3d83 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:36.850502 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb0b0b50-a999-4b3a-a54f-7771247b3d83 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:36.850775 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb0b0b50-a999-4b3a-a54f-7771247b3d83 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:36.851067 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb0b0b50-a999-4b3a-a54f-7771247b3d83 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:36.898772 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:36.919039 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb0b0b50-a999-4b3a-a54f-7771247b3d83 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:36.920422 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb0b0b50-a999-4b3a-a54f-7771247b3d83 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:36.998538 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1248d510-e048-442f-9ab5-15478d64d72d None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:36.998921 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1248d510-e048-442f-9ab5-15478d64d72d None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:36.999142 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1248d510-e048-442f-9ab5-15478d64d72d None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:37.000575 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 526/1040] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:00:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:00:37.005151 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74f360fb-148c-4a81-af37-50dbb9f84acb None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:37.005430 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74f360fb-148c-4a81-af37-50dbb9f84acb None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:37.005697 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74f360fb-148c-4a81-af37-50dbb9f84acb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:37.022620 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 515/1041] 10.222.0.77 () {58 vars in 1282 bytes} [Tue Aug 30 05:00:36 2022] GET /identity/v3/auth/tokens => generated 1969 bytes in 308 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 30 05:00:37.023711 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb0b0b50-a999-4b3a-a54f-7771247b3d83 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:37.029285 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-74f360fb-148c-4a81-af37-50dbb9f84acb None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:00:37.030535 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-74f360fb-148c-4a81-af37-50dbb9f84acb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:00:37.048601 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:37.098244 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 527/1042] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:00:37 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 30 05:00:37.108193 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:37.128461 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb0b0b50-a999-4b3a-a54f-7771247b3d83 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:37.162842 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11fd3ad2-d021-4c80-9442-f98dec966b1c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:37.163469 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11fd3ad2-d021-4c80-9442-f98dec966b1c None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:37.163746 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11fd3ad2-d021-4c80-9442-f98dec966b1c None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:37.164016 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11fd3ad2-d021-4c80-9442-f98dec966b1c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:37.173925 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11fd3ad2-d021-4c80-9442-f98dec966b1c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:37.175362 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11fd3ad2-d021-4c80-9442-f98dec966b1c None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:37.190225 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-11fd3ad2-d021-4c80-9442-f98dec966b1c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 05:00:37.190225 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:37.190225 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:00:37.190225 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:00:37.190225 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:00:37.190225 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:00:37.190225 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:00:37.190225 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:00:37.190225 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:00:37.190225 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:00:37.190225 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:00:37.190225 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:00:37.190225 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 05:00:37.190225 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 05:00:37.190225 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 05:00:37.190225 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 05:00:37.196673 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:00:37.196673 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:00:37.196673 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:00:37.196673 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:00:37.196673 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:00:37.196673 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:00:37.196673 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:00:37.196673 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 05:00:37.196673 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:00:37.196673 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:00:37.196673 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:00:37.196673 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:00:37.196673 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:00:37.196673 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:00:37.196673 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:00:37.196673 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:00:37.196673 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 05:00:37.197832 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 05:00:37.197832 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 05:00:37.197832 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:00:37.197832 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 516/1043] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 05:00:37 2022] GET /identity/v3/domains/Default => generated 87 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:00:37.198112 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74f360fb-148c-4a81-af37-50dbb9f84acb None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:37.296878 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74f360fb-148c-4a81-af37-50dbb9f84acb None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:37.332024 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6e06c38-40e7-4197-80db-df6e5f82f123 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:37.332618 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e06c38-40e7-4197-80db-df6e5f82f123 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:37.332895 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e06c38-40e7-4197-80db-df6e5f82f123 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:37.333162 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e06c38-40e7-4197-80db-df6e5f82f123 None admin] PATH_INFO: `/v3/domains` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:37.333793 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6e06c38-40e7-4197-80db-df6e5f82f123 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:37.334111 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6e06c38-40e7-4197-80db-df6e5f82f123 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:37.335494 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6e06c38-40e7-4197-80db-df6e5f82f123 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:37.351130 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 528/1044] 10.222.0.77 () {56 vars in 1109 bytes} [Tue Aug 30 05:00:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:37.352260 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11fd3ad2-d021-4c80-9442-f98dec966b1c None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:37.364387 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6e06c38-40e7-4197-80db-df6e5f82f123 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:37.382838 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:37.479672 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11fd3ad2-d021-4c80-9442-f98dec966b1c None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:37.480202 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6e06c38-40e7-4197-80db-df6e5f82f123 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:37.510320 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:37.516678 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8425d0b-e7c5-490d-af7f-9d4f097a6e5c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:37.516849 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19fe0459-c293-4e65-9601-39eec50d0f39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:37.517422 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8425d0b-e7c5-490d-af7f-9d4f097a6e5c None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:37.517524 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19fe0459-c293-4e65-9601-39eec50d0f39 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:37.517708 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8425d0b-e7c5-490d-af7f-9d4f097a6e5c None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:37.517805 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19fe0459-c293-4e65-9601-39eec50d0f39 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:37.517970 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8425d0b-e7c5-490d-af7f-9d4f097a6e5c None admin] PATH_INFO: `/v3/services/image` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:37.518066 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19fe0459-c293-4e65-9601-39eec50d0f39 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:37.518547 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8425d0b-e7c5-490d-af7f-9d4f097a6e5c None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:37.519947 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8425d0b-e7c5-490d-af7f-9d4f097a6e5c None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:37.525674 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19fe0459-c293-4e65-9601-39eec50d0f39 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:37.527132 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19fe0459-c293-4e65-9601-39eec50d0f39 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:37.529556 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-e8425d0b-e7c5-490d-af7f-9d4f097a6e5c None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Aug 30 05:00:37.529556 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:37.529556 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 30 05:00:37.529556 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 30 05:00:37.529556 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 30 05:00:37.529556 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 30 05:00:37.529556 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 30 05:00:37.529556 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 05:00:37.529556 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Aug 30 05:00:37.529556 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:00:37.529556 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 05:00:37.529556 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:00:37.529556 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:37.529556 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:00:37.529556 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:00:37.529556 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:00:37.529556 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:00:37.530710 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:00:37.530710 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:00:37.530710 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:00:37.530710 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:00:37.530710 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:00:37.530710 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:00:37.530710 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 30 05:00:37.530710 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 30 05:00:37.530710 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 30 05:00:37.530710 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 30 05:00:37.530710 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:00:37.530710 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:00:37.530710 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:00:37.530710 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:00:37.530710 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:00:37.530710 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:00:37.531680 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:00:37.531680 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 05:00:37.531680 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:00:37.531680 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:00:37.531680 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:00:37.531680 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:00:37.531680 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:00:37.531680 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:00:37.531680 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:00:37.531680 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:00:37.531680 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 30 05:00:37.531680 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 05:00:37.531680 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Aug 30 05:00:37.531680 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:00:37.531680 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 529/1045] 10.222.0.77 () {56 vars in 1105 bytes} [Tue Aug 30 05:00:37 2022] GET /identity/v3/services/image => generated 86 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 05:00:37.536606 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8425d0b-e7c5-490d-af7f-9d4f097a6e5c None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:37.541986 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-19fe0459-c293-4e65-9601-39eec50d0f39 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 05:00:37.541986 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:37.541986 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:00:37.541986 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:00:37.541986 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:00:37.541986 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:00:37.541986 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:00:37.541986 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:00:37.541986 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:00:37.541986 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:00:37.541986 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:00:37.541986 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:00:37.541986 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 05:00:37.541986 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 05:00:37.541986 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 05:00:37.542908 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 05:00:37.542908 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:00:37.542908 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:00:37.542908 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:00:37.542908 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:00:37.542908 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:00:37.542908 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:00:37.542908 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:00:37.542908 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 05:00:37.542908 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:00:37.542908 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:00:37.542908 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:00:37.542908 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:00:37.542908 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:00:37.542908 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:00:37.542908 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:00:37.542908 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:00:37.543801 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 05:00:37.543801 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 05:00:37.543801 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 05:00:37.543801 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 05:00:37.543801 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 05:00:37.543801 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 05:00:37.543801 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 05:00:37.543801 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:00:37.548014 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 517/1046] 10.222.0.77 () {56 vars in 1111 bytes} [Tue Aug 30 05:00:37 2022] GET /identity/v3/projects/service => generated 88 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 05:00:37.548667 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19fe0459-c293-4e65-9601-39eec50d0f39 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:37.620565 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:37.643104 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8425d0b-e7c5-490d-af7f-9d4f097a6e5c None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:37.654937 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19fe0459-c293-4e65-9601-39eec50d0f39 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:37.676561 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d27f373-7498-48bf-a8e8-c12b0973eed2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:37.677153 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d27f373-7498-48bf-a8e8-c12b0973eed2 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:37.677425 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d27f373-7498-48bf-a8e8-c12b0973eed2 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:37.677689 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d27f373-7498-48bf-a8e8-c12b0973eed2 None admin] PATH_INFO: `/v3/services` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:37.678299 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d27f373-7498-48bf-a8e8-c12b0973eed2 None admin] RBAC: Adding query filter params (name=image) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:37.678591 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d27f373-7498-48bf-a8e8-c12b0973eed2 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:37.679985 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d27f373-7498-48bf-a8e8-c12b0973eed2 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:37.684687 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 530/1047] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 05:00:37 2022] GET /identity/v3/services?name=image => generated 122 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:37.685871 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74915f8b-93f4-4191-891c-e7e4d62640f5 None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:37.686152 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74915f8b-93f4-4191-891c-e7e4d62640f5 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:37.686415 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74915f8b-93f4-4191-891c-e7e4d62640f5 None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:37.687311 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d538b6e9-a9ef-4129-b30e-3fa8abe014be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:37.687906 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d538b6e9-a9ef-4129-b30e-3fa8abe014be None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:37.688186 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d538b6e9-a9ef-4129-b30e-3fa8abe014be None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:37.688453 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d538b6e9-a9ef-4129-b30e-3fa8abe014be None admin] PATH_INFO: `/v3/projects` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:37.689136 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d538b6e9-a9ef-4129-b30e-3fa8abe014be None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:37.689513 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d538b6e9-a9ef-4129-b30e-3fa8abe014be None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:37.690940 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 531/1048] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:00:37 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:00:37.691060 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d538b6e9-a9ef-4129-b30e-3fa8abe014be None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:37.691868 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74915f8b-93f4-4191-891c-e7e4d62640f5 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:37.704132 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 518/1049] 10.222.0.77 () {56 vars in 1148 bytes} [Tue Aug 30 05:00:37 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:37.705355 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29f752c7-d42f-4ee4-8df5-0d7160abf36b None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:37.705635 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29f752c7-d42f-4ee4-8df5-0d7160abf36b None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:37.705901 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29f752c7-d42f-4ee4-8df5-0d7160abf36b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:37.723484 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-29f752c7-d42f-4ee4-8df5-0d7160abf36b None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:00:37.724590 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-29f752c7-d42f-4ee4-8df5-0d7160abf36b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:00:37.771431 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:37.792301 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 519/1050] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:00:37 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 05:00:37.792944 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29f752c7-d42f-4ee4-8df5-0d7160abf36b None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:37.803620 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74915f8b-93f4-4191-891c-e7e4d62640f5 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:37.840686 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28816dad-301e-4446-b0a2-4878a524ff50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:37.841392 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28816dad-301e-4446-b0a2-4878a524ff50 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:37.841726 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28816dad-301e-4446-b0a2-4878a524ff50 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:37.842056 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28816dad-301e-4446-b0a2-4878a524ff50 None admin] PATH_INFO: `/v3/services` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:37.842744 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28816dad-301e-4446-b0a2-4878a524ff50 None admin] RBAC: Adding query filter params (type=image) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:37.843147 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28816dad-301e-4446-b0a2-4878a524ff50 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:37.844564 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28816dad-301e-4446-b0a2-4878a524ff50 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:37.849912 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 532/1051] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 05:00:37 2022] GET /identity/v3/services?type=image => generated 350 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:37.855707 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28816dad-301e-4446-b0a2-4878a524ff50 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:37.864312 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:37.905894 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29f752c7-d42f-4ee4-8df5-0d7160abf36b None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:37.944552 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df59200f-75b7-4b81-bc68-11132a52339c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:37.945155 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df59200f-75b7-4b81-bc68-11132a52339c None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:37.945432 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df59200f-75b7-4b81-bc68-11132a52339c None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:37.945703 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df59200f-75b7-4b81-bc68-11132a52339c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:37.946535 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df59200f-75b7-4b81-bc68-11132a52339c None admin] RBAC: Adding query filter params (role.id=2d3c2095a28d4f4f8a8328c4b61b0015, scope.project.id=e9919141e3b947ab8d91b55bba4a7de6, user.id=886eb5e00bbf4ec292eb962b665b37d0) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:37.946863 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df59200f-75b7-4b81-bc68-11132a52339c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:37.948297 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df59200f-75b7-4b81-bc68-11132a52339c None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:37.962538 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28816dad-301e-4446-b0a2-4878a524ff50 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:37.964079 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 520/1052] 10.222.0.77 () {56 vars in 1374 bytes} [Tue Aug 30 05:00:37 2022] GET /identity/v3/role_assignments?user.id=886eb5e00bbf4ec292eb962b665b37d0&scope.project.id=e9919141e3b947ab8d91b55bba4a7de6&role.id=2d3c2095a28d4f4f8a8328c4b61b0015 => generated 259 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:38.001513 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80f8ed09-fcef-49a7-a70d-4e6cb8bb01be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:38.002046 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f8ed09-fcef-49a7-a70d-4e6cb8bb01be None admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:38.002322 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f8ed09-fcef-49a7-a70d-4e6cb8bb01be None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:38.002588 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f8ed09-fcef-49a7-a70d-4e6cb8bb01be None admin] PATH_INFO: `/v3/endpoints` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:38.003411 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80f8ed09-fcef-49a7-a70d-4e6cb8bb01be None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:38.004822 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80f8ed09-fcef-49a7-a70d-4e6cb8bb01be None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:38.028363 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 533/1053] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:00:37 2022] POST /identity/v3/endpoints => generated 327 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 05:00:38.061070 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df59200f-75b7-4b81-bc68-11132a52339c None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:38.116140 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:38.175910 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df59200f-75b7-4b81-bc68-11132a52339c None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:38.218942 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-736b0df1-01b6-4f13-a59b-ef298207c53f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:38.219568 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-736b0df1-01b6-4f13-a59b-ef298207c53f None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:38.219838 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-736b0df1-01b6-4f13-a59b-ef298207c53f None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:38.220100 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-736b0df1-01b6-4f13-a59b-ef298207c53f None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:38.232187 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-736b0df1-01b6-4f13-a59b-ef298207c53f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 05:00:38.236985 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-736b0df1-01b6-4f13-a59b-ef298207c53f None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 05:00:38.236985 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:38.236985 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:00:38.236985 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:00:38.236985 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:00:38.236985 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:00:38.236985 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:00:38.236985 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:00:38.236985 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:00:38.236985 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:00:38.236985 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:00:38.236985 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:00:38.236985 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 05:00:38.236985 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 05:00:38.236985 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 05:00:38.236985 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 05:00:38.237918 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 05:00:38.237918 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 05:00:38.237918 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 05:00:38.237918 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 05:00:38.237918 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:00:38.237918 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:00:38.237918 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:00:38.237918 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:00:38.237918 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:00:38.237918 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:00:38.237918 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:00:38.237918 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 05:00:38.237918 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:00:38.237918 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:00:38.237918 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:00:38.237918 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:00:38.238763 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:00:38.238763 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:00:38.238763 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:00:38.238763 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:00:38.238763 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 05:00:38.238763 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 05:00:38.238763 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 05:00:38.238763 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 05:00:38.238763 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 05:00:38.238763 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 05:00:38.238763 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 05:00:38.238763 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:00:38.238763 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 521/1054] 10.222.0.77 () {56 vars in 1096 bytes} [Tue Aug 30 05:00:38 2022] GET /identity/v3/roles/admin => generated 83 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:00:38.245263 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80f8ed09-fcef-49a7-a70d-4e6cb8bb01be None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:38.309971 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:38.351643 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80f8ed09-fcef-49a7-a70d-4e6cb8bb01be None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:38.385596 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6d7eaa6-a71b-4b6d-8bdd-6e82b8736341 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:38.386202 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6d7eaa6-a71b-4b6d-8bdd-6e82b8736341 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:38.386478 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6d7eaa6-a71b-4b6d-8bdd-6e82b8736341 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:38.386746 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6d7eaa6-a71b-4b6d-8bdd-6e82b8736341 None admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:38.387436 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6d7eaa6-a71b-4b6d-8bdd-6e82b8736341 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:38.387737 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6d7eaa6-a71b-4b6d-8bdd-6e82b8736341 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:38.389092 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6d7eaa6-a71b-4b6d-8bdd-6e82b8736341 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:38.398764 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 534/1055] 10.222.0.77 () {56 vars in 1099 bytes} [Tue Aug 30 05:00:38 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 05:00:38.404679 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-736b0df1-01b6-4f13-a59b-ef298207c53f None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:38.513228 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-736b0df1-01b6-4f13-a59b-ef298207c53f None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:38.515392 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:38.550265 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-113a82b4-bbb3-43b6-83cc-053f58bf88d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:38.550938 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-113a82b4-bbb3-43b6-83cc-053f58bf88d9 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:38.551207 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-113a82b4-bbb3-43b6-83cc-053f58bf88d9 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:38.551477 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-113a82b4-bbb3-43b6-83cc-053f58bf88d9 None admin] PATH_INFO: `/v3/users/c4f1c3280e864ff1bc823775ea5ea3a5` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:38.565810 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-113a82b4-bbb3-43b6-83cc-053f58bf88d9 None admin] RBAC: Authorizing `identity:get_user(user_id=c4f1c3280e864ff1bc823775ea5ea3a5)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:38.567224 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-113a82b4-bbb3-43b6-83cc-053f58bf88d9 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:38.570894 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 522/1056] 10.222.0.77 () {56 vars in 1177 bytes} [Tue Aug 30 05:00:38 2022] GET /identity/v3/users/c4f1c3280e864ff1bc823775ea5ea3a5 => generated 285 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:38.574800 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6d7eaa6-a71b-4b6d-8bdd-6e82b8736341 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:38.675903 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6d7eaa6-a71b-4b6d-8bdd-6e82b8736341 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:38.698873 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:38.711250 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-704d6873-4788-4dce-99e1-6f808276f75d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:38.711854 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-704d6873-4788-4dce-99e1-6f808276f75d None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:38.712132 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-704d6873-4788-4dce-99e1-6f808276f75d None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:38.712400 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-704d6873-4788-4dce-99e1-6f808276f75d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:38.779457 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-704d6873-4788-4dce-99e1-6f808276f75d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:38.780903 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-704d6873-4788-4dce-99e1-6f808276f75d None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:38.877863 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 535/1057] 10.222.0.77 () {58 vars in 1282 bytes} [Tue Aug 30 05:00:38 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 303 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 30 05:00:38.883607 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-113a82b4-bbb3-43b6-83cc-053f58bf88d9 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:38.988392 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-113a82b4-bbb3-43b6-83cc-053f58bf88d9 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:39.022711 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a4a630d-c017-4eec-9263-ff5ffbe934e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:39.023397 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a4a630d-c017-4eec-9263-ff5ffbe934e6 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:39.023672 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a4a630d-c017-4eec-9263-ff5ffbe934e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:39.023948 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a4a630d-c017-4eec-9263-ff5ffbe934e6 None admin] PATH_INFO: `/v3/projects/c676bac7a20f42ce88c6f9805903d625` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:39.025146 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a4a630d-c017-4eec-9263-ff5ffbe934e6 None admin] RBAC: Authorizing `identity:get_project(project_id=c676bac7a20f42ce88c6f9805903d625)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:39.026608 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a4a630d-c017-4eec-9263-ff5ffbe934e6 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:39.028548 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 523/1058] 10.222.0.77 () {56 vars in 1186 bytes} [Tue Aug 30 05:00:38 2022] GET /identity/v3/projects/c676bac7a20f42ce88c6f9805903d625 => generated 311 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:39.034761 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-704d6873-4788-4dce-99e1-6f808276f75d None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:39.059927 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:39.136927 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-704d6873-4788-4dce-99e1-6f808276f75d None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:39.173187 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-636d5614-3c05-4fbc-8493-603e3adbfc70 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:39.173776 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-636d5614-3c05-4fbc-8493-603e3adbfc70 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:39.174046 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-636d5614-3c05-4fbc-8493-603e3adbfc70 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:39.174312 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-636d5614-3c05-4fbc-8493-603e3adbfc70 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:39.175108 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-636d5614-3c05-4fbc-8493-603e3adbfc70 None admin] RBAC: Adding query filter params (role.id=9d9d5d2c270146488249b7ed943280db, scope.project.id=c676bac7a20f42ce88c6f9805903d625, user.id=c4f1c3280e864ff1bc823775ea5ea3a5) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:39.175408 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-636d5614-3c05-4fbc-8493-603e3adbfc70 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:39.176802 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-636d5614-3c05-4fbc-8493-603e3adbfc70 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:39.182283 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 536/1059] 10.222.0.77 () {56 vars in 1374 bytes} [Tue Aug 30 05:00:39 2022] GET /identity/v3/role_assignments?user.id=c4f1c3280e864ff1bc823775ea5ea3a5&scope.project.id=c676bac7a20f42ce88c6f9805903d625&role.id=9d9d5d2c270146488249b7ed943280db => generated 259 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:39.863393 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21d061ac-e5f4-4af9-bf50-2068a3727216 None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:39.863932 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21d061ac-e5f4-4af9-bf50-2068a3727216 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:39.863932 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21d061ac-e5f4-4af9-bf50-2068a3727216 None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:39.865433 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 524/1060] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:00:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:00:39.871368 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-238e9ff1-a29f-4117-97de-212c7d81f039 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:39.871798 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-238e9ff1-a29f-4117-97de-212c7d81f039 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:39.871963 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-238e9ff1-a29f-4117-97de-212c7d81f039 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:39.889954 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-238e9ff1-a29f-4117-97de-212c7d81f039 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:00:39.891175 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-238e9ff1-a29f-4117-97de-212c7d81f039 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:00:39.961240 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 537/1061] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:00:39 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 05:00:39.972615 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a84f89f4-1806-452a-848a-6af369d03216 None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:39.972898 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a84f89f4-1806-452a-848a-6af369d03216 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:39.973166 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a84f89f4-1806-452a-848a-6af369d03216 None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:39.974624 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 525/1062] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:00:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:00:39.980234 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-526f7618-0898-4f38-909b-17e4e01b43e8 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:39.980518 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-526f7618-0898-4f38-909b-17e4e01b43e8 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:39.980788 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-526f7618-0898-4f38-909b-17e4e01b43e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:39.999119 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-526f7618-0898-4f38-909b-17e4e01b43e8 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:00:40.000284 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-526f7618-0898-4f38-909b-17e4e01b43e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:00:40.013430 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:40.067417 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 538/1063] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:00:39 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 05:00:40.230660 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a84f89f4-1806-452a-848a-6af369d03216 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:40.258703 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:40.328318 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-526f7618-0898-4f38-909b-17e4e01b43e8 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:40.332414 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a84f89f4-1806-452a-848a-6af369d03216 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:40.369478 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-595aa1e3-402a-4704-aca2-4db2308b926b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:40.370218 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-595aa1e3-402a-4704-aca2-4db2308b926b None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:40.370501 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-595aa1e3-402a-4704-aca2-4db2308b926b None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:40.370781 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-595aa1e3-402a-4704-aca2-4db2308b926b None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:40.381743 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-595aa1e3-402a-4704-aca2-4db2308b926b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 05:00:40.387078 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-595aa1e3-402a-4704-aca2-4db2308b926b None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 30 05:00:40.387078 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:40.387078 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:00:40.387078 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:00:40.387078 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:00:40.387078 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:00:40.387078 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:00:40.387078 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:00:40.387078 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:00:40.387078 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:00:40.387078 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:00:40.387078 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:00:40.387078 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 05:00:40.387078 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 05:00:40.387078 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 05:00:40.387078 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 05:00:40.387988 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 05:00:40.387988 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 05:00:40.387988 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 05:00:40.387988 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 05:00:40.387988 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:00:40.387988 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:00:40.387988 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:00:40.387988 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:00:40.387988 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:00:40.387988 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:00:40.387988 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:00:40.387988 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 05:00:40.387988 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:00:40.387988 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:00:40.387988 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:00:40.387988 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:00:40.388824 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:00:40.388824 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:00:40.388824 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:00:40.388824 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:00:40.388824 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 05:00:40.388824 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 05:00:40.388824 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 05:00:40.388824 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 05:00:40.388824 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 05:00:40.388824 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 05:00:40.388824 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 30 05:00:40.388824 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:00:40.388824 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 526/1064] 10.222.0.77 () {56 vars in 1120 bytes} [Tue Aug 30 05:00:40 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:00:40.394058 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-595aa1e3-402a-4704-aca2-4db2308b926b None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:40.431411 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-526f7618-0898-4f38-909b-17e4e01b43e8 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:40.467523 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d2a0692-4cd2-43be-b5d7-e66c77ef2a51 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:40.468336 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d2a0692-4cd2-43be-b5d7-e66c77ef2a51 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:40.468617 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d2a0692-4cd2-43be-b5d7-e66c77ef2a51 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:40.468885 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d2a0692-4cd2-43be-b5d7-e66c77ef2a51 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:40.488865 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:40.495584 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:40.508081 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-595aa1e3-402a-4704-aca2-4db2308b926b None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:40.544119 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d2a0692-4cd2-43be-b5d7-e66c77ef2a51 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:40.545106 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a806f11-ea42-4a2e-87cb-873ff644e57c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:40.545567 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d2a0692-4cd2-43be-b5d7-e66c77ef2a51 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:40.545734 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a806f11-ea42-4a2e-87cb-873ff644e57c None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:40.545975 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a806f11-ea42-4a2e-87cb-873ff644e57c None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:40.546242 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a806f11-ea42-4a2e-87cb-873ff644e57c None admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:40.546918 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a806f11-ea42-4a2e-87cb-873ff644e57c None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:40.547300 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a806f11-ea42-4a2e-87cb-873ff644e57c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:40.548584 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a806f11-ea42-4a2e-87cb-873ff644e57c None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:40.562280 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 527/1065] 10.222.0.77 () {56 vars in 1115 bytes} [Tue Aug 30 05:00:40 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 05:00:40.563517 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a806f11-ea42-4a2e-87cb-873ff644e57c None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:40.649282 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 539/1066] 10.222.0.77 () {58 vars in 1282 bytes} [Tue Aug 30 05:00:40 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 321 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Aug 30 05:00:40.654604 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d2a0692-4cd2-43be-b5d7-e66c77ef2a51 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:40.675678 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a806f11-ea42-4a2e-87cb-873ff644e57c None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:40.703430 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:40.711800 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da80c8fc-8132-489f-89a7-b3551c53ac9c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:40.712985 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da80c8fc-8132-489f-89a7-b3551c53ac9c None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:40.713275 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da80c8fc-8132-489f-89a7-b3551c53ac9c None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:40.713541 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da80c8fc-8132-489f-89a7-b3551c53ac9c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:40.727904 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da80c8fc-8132-489f-89a7-b3551c53ac9c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:40.729363 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da80c8fc-8132-489f-89a7-b3551c53ac9c None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:40.747057 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-da80c8fc-8132-489f-89a7-b3551c53ac9c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 05:00:40.747057 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:40.747057 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:00:40.747057 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:00:40.747057 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:00:40.747057 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:00:40.747057 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:00:40.747057 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:00:40.747057 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:00:40.747057 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:00:40.747057 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:00:40.747057 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:00:40.747057 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 05:00:40.747057 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 05:00:40.747057 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 05:00:40.747057 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 05:00:40.748046 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:00:40.748046 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:00:40.748046 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:00:40.748046 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:00:40.748046 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:00:40.748046 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:00:40.748046 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:00:40.748046 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 05:00:40.748046 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:00:40.748046 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:00:40.748046 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:00:40.748046 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:00:40.748046 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:00:40.748046 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:00:40.748046 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:00:40.748046 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:00:40.748046 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 05:00:40.748930 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 05:00:40.748930 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 05:00:40.748930 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:00:40.748930 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 528/1067] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 05:00:40 2022] GET /identity/v3/domains/Default => generated 87 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:00:40.754191 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da80c8fc-8132-489f-89a7-b3551c53ac9c None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:40.758768 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:40.761125 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d2a0692-4cd2-43be-b5d7-e66c77ef2a51 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:40.801741 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1db9969-f76f-42be-a280-d35a292025be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:40.802369 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1db9969-f76f-42be-a280-d35a292025be None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:40.802643 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1db9969-f76f-42be-a280-d35a292025be None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:40.802963 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1db9969-f76f-42be-a280-d35a292025be None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:40.813136 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1db9969-f76f-42be-a280-d35a292025be None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:40.814578 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1db9969-f76f-42be-a280-d35a292025be None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:40.829973 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-a1db9969-f76f-42be-a280-d35a292025be None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 05:00:40.829973 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:40.829973 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:00:40.829973 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:00:40.829973 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:00:40.829973 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:00:40.829973 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:00:40.829973 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:00:40.829973 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:00:40.829973 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:00:40.829973 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:00:40.829973 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:00:40.829973 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 05:00:40.829973 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 05:00:40.829973 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 05:00:40.829973 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 05:00:40.830962 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:00:40.830962 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:00:40.830962 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:00:40.830962 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:00:40.830962 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:00:40.830962 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:00:40.830962 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:00:40.830962 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 05:00:40.830962 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:00:40.830962 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:00:40.830962 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:00:40.830962 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:00:40.830962 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:00:40.830962 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:00:40.830962 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:00:40.830962 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:00:40.830962 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 05:00:40.831873 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 05:00:40.831873 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 05:00:40.831873 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:00:40.831873 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 540/1068] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 05:00:40 2022] GET /identity/v3/domains/Default => generated 87 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 05:00:40.837875 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1db9969-f76f-42be-a280-d35a292025be None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:40.872603 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da80c8fc-8132-489f-89a7-b3551c53ac9c None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:40.909419 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b57a6d40-c468-4fe6-aa56-bc7a894cf51a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:40.910435 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b57a6d40-c468-4fe6-aa56-bc7a894cf51a None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:40.910993 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b57a6d40-c468-4fe6-aa56-bc7a894cf51a None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:40.911488 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b57a6d40-c468-4fe6-aa56-bc7a894cf51a None admin] PATH_INFO: `/v3/domains` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:40.912293 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b57a6d40-c468-4fe6-aa56-bc7a894cf51a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:40.912754 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b57a6d40-c468-4fe6-aa56-bc7a894cf51a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:40.914382 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b57a6d40-c468-4fe6-aa56-bc7a894cf51a None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:40.927565 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 529/1069] 10.222.0.77 () {56 vars in 1109 bytes} [Tue Aug 30 05:00:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 05:00:40.933597 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b57a6d40-c468-4fe6-aa56-bc7a894cf51a None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:40.953536 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:40.958994 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1db9969-f76f-42be-a280-d35a292025be None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:40.996402 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:40.997347 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a640d530-33e9-4296-ae20-83bdba93b48c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:40.997996 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a640d530-33e9-4296-ae20-83bdba93b48c None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:40.998350 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a640d530-33e9-4296-ae20-83bdba93b48c None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:40.998633 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a640d530-33e9-4296-ae20-83bdba93b48c None admin] PATH_INFO: `/v3/domains` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:40.999533 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a640d530-33e9-4296-ae20-83bdba93b48c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:40.999843 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a640d530-33e9-4296-ae20-83bdba93b48c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:41.001217 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a640d530-33e9-4296-ae20-83bdba93b48c None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:41.013994 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 541/1070] 10.222.0.77 () {56 vars in 1109 bytes} [Tue Aug 30 05:00:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 05:00:41.047497 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b57a6d40-c468-4fe6-aa56-bc7a894cf51a None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:41.057074 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-124c4fe3-6741-4e9b-a065-6856d3e17d6e None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:41.057410 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-124c4fe3-6741-4e9b-a065-6856d3e17d6e None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:41.057628 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-124c4fe3-6741-4e9b-a065-6856d3e17d6e None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:41.059115 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 542/1071] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:00:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 05:00:41.064966 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02a4114e-34ad-4152-bf65-6ea88ef0930d None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:41.065252 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02a4114e-34ad-4152-bf65-6ea88ef0930d None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:41.065565 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02a4114e-34ad-4152-bf65-6ea88ef0930d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:41.084696 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-02a4114e-34ad-4152-bf65-6ea88ef0930d None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:00:41.085937 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-02a4114e-34ad-4152-bf65-6ea88ef0930d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:00:41.088209 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19844345-5cfd-43e8-b65b-a76f91e6000b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:41.088852 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19844345-5cfd-43e8-b65b-a76f91e6000b None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:41.089135 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19844345-5cfd-43e8-b65b-a76f91e6000b None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:41.089406 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19844345-5cfd-43e8-b65b-a76f91e6000b None admin] PATH_INFO: `/v3/users/nova` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:41.097638 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19844345-5cfd-43e8-b65b-a76f91e6000b None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:41.099143 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19844345-5cfd-43e8-b65b-a76f91e6000b None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:41.112678 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-19844345-5cfd-43e8-b65b-a76f91e6000b None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 30 05:00:41.112678 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:41.112678 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:00:41.112678 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:00:41.112678 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:00:41.112678 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:00:41.112678 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:00:41.112678 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:00:41.112678 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:00:41.112678 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:00:41.112678 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:00:41.112678 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:00:41.112678 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 05:00:41.112678 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 05:00:41.112678 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 05:00:41.112678 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 05:00:41.113759 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:00:41.113759 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:00:41.113759 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 05:00:41.113759 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 05:00:41.113759 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 05:00:41.113759 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 05:00:41.113759 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:00:41.113759 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:00:41.113759 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:00:41.113759 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:00:41.113759 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:00:41.113759 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 05:00:41.113759 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:00:41.113759 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:00:41.113759 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:00:41.113759 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:00:41.113759 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:00:41.113759 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:00:41.116397 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:00:41.116397 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:00:41.116397 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 05:00:41.116397 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 05:00:41.116397 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 05:00:41.116397 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 05:00:41.116397 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 05:00:41.116397 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 05:00:41.116397 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 30 05:00:41.116397 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:00:41.116397 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 530/1072] 10.222.0.77 () {56 vars in 1093 bytes} [Tue Aug 30 05:00:40 2022] GET /identity/v3/users/nova => generated 82 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 05:00:41.120423 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19844345-5cfd-43e8-b65b-a76f91e6000b None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:41.166257 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 543/1073] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:00:41 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 30 05:00:41.230692 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19844345-5cfd-43e8-b65b-a76f91e6000b None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:41.253248 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:41.269149 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07ea8c5b-3efd-4c55-bae2-b8de59595ecd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:41.269759 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ea8c5b-3efd-4c55-bae2-b8de59595ecd None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:41.270040 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ea8c5b-3efd-4c55-bae2-b8de59595ecd None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:41.270313 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ea8c5b-3efd-4c55-bae2-b8de59595ecd None admin] PATH_INFO: `/v3/users` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:41.271168 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07ea8c5b-3efd-4c55-bae2-b8de59595ecd None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:41.271503 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07ea8c5b-3efd-4c55-bae2-b8de59595ecd None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:41.272935 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07ea8c5b-3efd-4c55-bae2-b8de59595ecd None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:41.291564 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 531/1074] 10.222.0.77 () {56 vars in 1133 bytes} [Tue Aug 30 05:00:41 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 05:00:41.297362 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02a4114e-34ad-4152-bf65-6ea88ef0930d None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:41.395034 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:41.404884 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02a4114e-34ad-4152-bf65-6ea88ef0930d None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:41.433527 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07ea8c5b-3efd-4c55-bae2-b8de59595ecd None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:41.438632 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd24919c-54f2-4fc0-9114-eb22ce165fd6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:41.439289 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd24919c-54f2-4fc0-9114-eb22ce165fd6 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:41.439566 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd24919c-54f2-4fc0-9114-eb22ce165fd6 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:41.439831 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd24919c-54f2-4fc0-9114-eb22ce165fd6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:41.449698 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd24919c-54f2-4fc0-9114-eb22ce165fd6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:41.451164 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd24919c-54f2-4fc0-9114-eb22ce165fd6 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:41.471764 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-bd24919c-54f2-4fc0-9114-eb22ce165fd6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 05:00:41.471764 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:41.471764 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:00:41.471764 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:00:41.471764 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:00:41.471764 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:00:41.471764 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:00:41.471764 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:00:41.471764 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:00:41.471764 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:00:41.471764 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:00:41.471764 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:00:41.471764 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 05:00:41.471764 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 05:00:41.471764 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 05:00:41.471764 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 05:00:41.472707 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:00:41.472707 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:00:41.472707 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:00:41.472707 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:00:41.472707 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:00:41.472707 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:00:41.472707 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:00:41.472707 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 05:00:41.472707 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:00:41.472707 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:00:41.472707 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:00:41.472707 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:00:41.472707 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:00:41.472707 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:00:41.472707 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:00:41.472707 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:00:41.472707 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 05:00:41.473669 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 05:00:41.473669 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 05:00:41.473669 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:00:41.473669 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 544/1075] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 05:00:41 2022] GET /identity/v3/domains/Default => generated 87 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:00:41.479359 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd24919c-54f2-4fc0-9114-eb22ce165fd6 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:41.539132 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07ea8c5b-3efd-4c55-bae2-b8de59595ecd None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:41.571982 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8349a8d2-592f-454a-9121-c2598be41a92 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:41.572596 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8349a8d2-592f-454a-9121-c2598be41a92 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:41.572875 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8349a8d2-592f-454a-9121-c2598be41a92 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:41.573145 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8349a8d2-592f-454a-9121-c2598be41a92 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:41.576165 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd24919c-54f2-4fc0-9114-eb22ce165fd6 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:41.582888 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8349a8d2-592f-454a-9121-c2598be41a92 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 05:00:41.587677 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-8349a8d2-592f-454a-9121-c2598be41a92 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 05:00:41.587677 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:41.587677 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:00:41.587677 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:00:41.587677 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:00:41.587677 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:00:41.587677 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:00:41.587677 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:00:41.587677 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:00:41.587677 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:00:41.587677 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:00:41.587677 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:00:41.587677 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 05:00:41.587677 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 05:00:41.587677 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 05:00:41.587677 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 05:00:41.588722 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 05:00:41.588722 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 05:00:41.588722 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 05:00:41.588722 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 05:00:41.588722 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:00:41.588722 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:00:41.588722 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:00:41.588722 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:00:41.588722 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:00:41.588722 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:00:41.588722 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:00:41.588722 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 05:00:41.588722 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:00:41.588722 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:00:41.588722 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:00:41.588722 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:00:41.589574 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:00:41.589574 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:00:41.589574 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:00:41.589574 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:00:41.589574 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 05:00:41.589574 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 05:00:41.589574 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 05:00:41.589574 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 05:00:41.589574 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 05:00:41.589574 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 05:00:41.589574 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 05:00:41.589574 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:00:41.589574 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 532/1076] 10.222.0.77 () {56 vars in 1096 bytes} [Tue Aug 30 05:00:41 2022] GET /identity/v3/roles/admin => generated 83 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 05:00:41.594595 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8349a8d2-592f-454a-9121-c2598be41a92 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:41.611558 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e4a4c85-9d28-4dec-bf45-bb4d743837b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:41.612161 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e4a4c85-9d28-4dec-bf45-bb4d743837b8 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:41.612442 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e4a4c85-9d28-4dec-bf45-bb4d743837b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:41.612715 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e4a4c85-9d28-4dec-bf45-bb4d743837b8 None admin] PATH_INFO: `/v3/domains` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:41.613352 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e4a4c85-9d28-4dec-bf45-bb4d743837b8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:41.613652 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e4a4c85-9d28-4dec-bf45-bb4d743837b8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:41.615065 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e4a4c85-9d28-4dec-bf45-bb4d743837b8 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:41.627909 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 545/1077] 10.222.0.77 () {56 vars in 1109 bytes} [Tue Aug 30 05:00:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 05:00:41.633612 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e4a4c85-9d28-4dec-bf45-bb4d743837b8 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:41.636194 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:41.639452 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:41.696821 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8349a8d2-592f-454a-9121-c2598be41a92 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:41.729738 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e4a4c85-9d28-4dec-bf45-bb4d743837b8 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:41.730626 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38cf3e30-5f09-4f39-8e91-39124464869e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:41.731254 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38cf3e30-5f09-4f39-8e91-39124464869e None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:41.731534 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38cf3e30-5f09-4f39-8e91-39124464869e None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:41.731799 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38cf3e30-5f09-4f39-8e91-39124464869e None admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:41.732469 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38cf3e30-5f09-4f39-8e91-39124464869e None admin] RBAC: Adding query filter params (name=admin) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:41.732771 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38cf3e30-5f09-4f39-8e91-39124464869e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:41.734133 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38cf3e30-5f09-4f39-8e91-39124464869e None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:41.742406 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 533/1078] 10.222.0.77 () {56 vars in 1099 bytes} [Tue Aug 30 05:00:41 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 05:00:41.748100 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38cf3e30-5f09-4f39-8e91-39124464869e None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:41.762902 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3afd825a-ef92-4cc1-9710-22c0f9c5f9a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:41.763514 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3afd825a-ef92-4cc1-9710-22c0f9c5f9a4 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:41.763785 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3afd825a-ef92-4cc1-9710-22c0f9c5f9a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:41.764053 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3afd825a-ef92-4cc1-9710-22c0f9c5f9a4 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:41.770503 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3afd825a-ef92-4cc1-9710-22c0f9c5f9a4 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:41.772509 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3afd825a-ef92-4cc1-9710-22c0f9c5f9a4 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:41.783306 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-3afd825a-ef92-4cc1-9710-22c0f9c5f9a4 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 05:00:41.783306 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:41.783306 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:00:41.783306 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:00:41.783306 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:00:41.783306 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:00:41.783306 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:00:41.783306 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:00:41.783306 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:00:41.783306 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:00:41.783306 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:00:41.783306 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:00:41.783306 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 05:00:41.783306 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 05:00:41.783306 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 05:00:41.784377 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 05:00:41.784377 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:00:41.784377 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:00:41.784377 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:00:41.784377 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:00:41.784377 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:00:41.784377 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:00:41.784377 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:00:41.784377 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 05:00:41.784377 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:00:41.784377 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:00:41.784377 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:00:41.784377 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:00:41.784377 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:00:41.784377 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:00:41.784377 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:00:41.784377 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:00:41.785436 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 05:00:41.785436 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 05:00:41.785436 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 05:00:41.785436 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 05:00:41.785436 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 05:00:41.785436 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 05:00:41.785436 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 05:00:41.785436 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:00:41.785436 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 546/1079] 10.222.0.77 () {56 vars in 1111 bytes} [Tue Aug 30 05:00:41 2022] GET /identity/v3/projects/service => generated 88 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 05:00:41.790751 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3afd825a-ef92-4cc1-9710-22c0f9c5f9a4 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:41.847624 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38cf3e30-5f09-4f39-8e91-39124464869e None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:41.849049 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:41.851668 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:41.881392 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ff0c100-4648-47c7-a2d1-3cac7e0d4218 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:41.882002 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff0c100-4648-47c7-a2d1-3cac7e0d4218 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:41.882276 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff0c100-4648-47c7-a2d1-3cac7e0d4218 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:41.882541 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff0c100-4648-47c7-a2d1-3cac7e0d4218 None admin] PATH_INFO: `/v3/users/c4f1c3280e864ff1bc823775ea5ea3a5` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:41.883683 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ff0c100-4648-47c7-a2d1-3cac7e0d4218 None admin] RBAC: Authorizing `identity:get_user(user_id=c4f1c3280e864ff1bc823775ea5ea3a5)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:41.885087 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ff0c100-4648-47c7-a2d1-3cac7e0d4218 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:41.886667 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 534/1080] 10.222.0.77 () {56 vars in 1177 bytes} [Tue Aug 30 05:00:41 2022] GET /identity/v3/users/c4f1c3280e864ff1bc823775ea5ea3a5 => generated 285 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 05:00:41.889960 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3afd825a-ef92-4cc1-9710-22c0f9c5f9a4 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:41.892263 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ff0c100-4648-47c7-a2d1-3cac7e0d4218 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:41.925139 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea52ad81-b53f-4c3a-ac94-3048137d8edc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:41.925736 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea52ad81-b53f-4c3a-ac94-3048137d8edc None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:41.926016 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea52ad81-b53f-4c3a-ac94-3048137d8edc None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:41.926283 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea52ad81-b53f-4c3a-ac94-3048137d8edc None admin] PATH_INFO: `/v3/projects` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:41.926996 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea52ad81-b53f-4c3a-ac94-3048137d8edc None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:41.927298 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea52ad81-b53f-4c3a-ac94-3048137d8edc None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:41.928718 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea52ad81-b53f-4c3a-ac94-3048137d8edc None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:41.940940 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 547/1081] 10.222.0.77 () {56 vars in 1148 bytes} [Tue Aug 30 05:00:41 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 05:00:41.946674 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea52ad81-b53f-4c3a-ac94-3048137d8edc None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:41.992137 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ff0c100-4648-47c7-a2d1-3cac7e0d4218 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:42.025439 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c135c15a-1eae-41e6-8a03-346af0282a9b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:42.026064 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c135c15a-1eae-41e6-8a03-346af0282a9b None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:42.026341 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c135c15a-1eae-41e6-8a03-346af0282a9b None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:42.026614 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c135c15a-1eae-41e6-8a03-346af0282a9b None admin] PATH_INFO: `/v3/projects/c676bac7a20f42ce88c6f9805903d625` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:42.027748 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c135c15a-1eae-41e6-8a03-346af0282a9b None admin] RBAC: Authorizing `identity:get_project(project_id=c676bac7a20f42ce88c6f9805903d625)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:42.029176 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c135c15a-1eae-41e6-8a03-346af0282a9b None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:42.030791 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 535/1082] 10.222.0.77 () {56 vars in 1186 bytes} [Tue Aug 30 05:00:41 2022] GET /identity/v3/projects/c676bac7a20f42ce88c6f9805903d625 => generated 311 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 05:00:42.036673 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c135c15a-1eae-41e6-8a03-346af0282a9b None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:42.047339 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea52ad81-b53f-4c3a-ac94-3048137d8edc None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:42.071164 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:42.076333 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:42.084071 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab04b8f8-65de-4d51-a8cd-8a0fb9949447 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:42.085509 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab04b8f8-65de-4d51-a8cd-8a0fb9949447 None admin] REQUEST_METHOD: `PUT` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:42.085788 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab04b8f8-65de-4d51-a8cd-8a0fb9949447 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:42.086061 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab04b8f8-65de-4d51-a8cd-8a0fb9949447 None admin] PATH_INFO: `/v3/projects/e9919141e3b947ab8d91b55bba4a7de6/users/886eb5e00bbf4ec292eb962b665b37d0/roles/2d3c2095a28d4f4f8a8328c4b61b0015` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:42.088620 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab04b8f8-65de-4d51-a8cd-8a0fb9949447 None admin] RBAC: Authorizing `identity:create_grant(project_id=e9919141e3b947ab8d91b55bba4a7de6, user_id=886eb5e00bbf4ec292eb962b665b37d0, role_id=2d3c2095a28d4f4f8a8328c4b61b0015)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:42.090219 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab04b8f8-65de-4d51-a8cd-8a0fb9949447 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:42.104153 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 548/1083] 10.222.0.77 () {58 vars in 1439 bytes} [Tue Aug 30 05:00:41 2022] PUT /identity/v3/projects/e9919141e3b947ab8d91b55bba4a7de6/users/886eb5e00bbf4ec292eb962b665b37d0/roles/2d3c2095a28d4f4f8a8328c4b61b0015 => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 05:00:42.140669 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c135c15a-1eae-41e6-8a03-346af0282a9b None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:42.176683 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-947f6c4f-2ec2-4b0f-8821-1ed681182b9c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:42.177357 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-947f6c4f-2ec2-4b0f-8821-1ed681182b9c None admin] REQUEST_METHOD: `PUT` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:42.177639 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-947f6c4f-2ec2-4b0f-8821-1ed681182b9c None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:42.177963 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-947f6c4f-2ec2-4b0f-8821-1ed681182b9c None admin] PATH_INFO: `/v3/projects/c676bac7a20f42ce88c6f9805903d625/users/c4f1c3280e864ff1bc823775ea5ea3a5/roles/9d9d5d2c270146488249b7ed943280db` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:42.179964 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-947f6c4f-2ec2-4b0f-8821-1ed681182b9c None admin] RBAC: Authorizing `identity:create_grant(project_id=c676bac7a20f42ce88c6f9805903d625, user_id=c4f1c3280e864ff1bc823775ea5ea3a5, role_id=9d9d5d2c270146488249b7ed943280db)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:42.181411 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-947f6c4f-2ec2-4b0f-8821-1ed681182b9c None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:42.195891 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 536/1084] 10.222.0.77 () {58 vars in 1439 bytes} [Tue Aug 30 05:00:42 2022] PUT /identity/v3/projects/c676bac7a20f42ce88c6f9805903d625/users/c4f1c3280e864ff1bc823775ea5ea3a5/roles/9d9d5d2c270146488249b7ed943280db => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 05:00:43.993621 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46024a7c-8e0a-4bba-b206-27aaaa323a23 None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:43.994214 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46024a7c-8e0a-4bba-b206-27aaaa323a23 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:43.994214 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46024a7c-8e0a-4bba-b206-27aaaa323a23 None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:43.995691 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 549/1085] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:00:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:00:44.000632 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f846069-5a2b-401f-bff4-71496f535ab8 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:44.001037 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f846069-5a2b-401f-bff4-71496f535ab8 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:44.001147 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f846069-5a2b-401f-bff4-71496f535ab8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:44.018809 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-8f846069-5a2b-401f-bff4-71496f535ab8 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:00:44.020004 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-8f846069-5a2b-401f-bff4-71496f535ab8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:00:44.074586 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1ab66f6-3d2a-477b-9d36-195af8e424e0 None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:44.074906 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1ab66f6-3d2a-477b-9d36-195af8e424e0 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:44.075181 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1ab66f6-3d2a-477b-9d36-195af8e424e0 None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:44.076504 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 550/1086] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:00:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 05:00:44.081231 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77a75f3f-3ecf-4483-8500-b370bc1226ee None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:44.081509 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77a75f3f-3ecf-4483-8500-b370bc1226ee None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:44.081777 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77a75f3f-3ecf-4483-8500-b370bc1226ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:44.088899 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 537/1087] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:00:43 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 05:00:44.099455 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-77a75f3f-3ecf-4483-8500-b370bc1226ee None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:00:44.101151 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-77a75f3f-3ecf-4483-8500-b370bc1226ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:00:44.165446 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 551/1088] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:00:44 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 30 05:00:44.354197 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f846069-5a2b-401f-bff4-71496f535ab8 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:44.381022 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:44.433950 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77a75f3f-3ecf-4483-8500-b370bc1226ee None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:44.452591 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f846069-5a2b-401f-bff4-71496f535ab8 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:44.489378 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24e34f02-701c-4232-879c-ea21045b63f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:44.489999 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e34f02-701c-4232-879c-ea21045b63f7 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:44.490303 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e34f02-701c-4232-879c-ea21045b63f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:44.490575 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e34f02-701c-4232-879c-ea21045b63f7 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:44.500507 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24e34f02-701c-4232-879c-ea21045b63f7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 05:00:44.505254 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-24e34f02-701c-4232-879c-ea21045b63f7 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 30 05:00:44.505254 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:44.505254 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:00:44.505254 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:00:44.505254 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:00:44.505254 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:00:44.505254 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:00:44.505254 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:00:44.505254 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:00:44.505254 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:00:44.505254 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:00:44.505254 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:00:44.505254 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 05:00:44.505254 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 05:00:44.505254 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 05:00:44.505254 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 05:00:44.506397 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 05:00:44.506397 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 05:00:44.506397 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 05:00:44.506397 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 05:00:44.506397 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:00:44.506397 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:00:44.506397 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:00:44.506397 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:00:44.506397 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:00:44.506397 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:00:44.506397 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:00:44.506397 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 05:00:44.506397 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:00:44.506397 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:00:44.506397 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:00:44.506397 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:00:44.507282 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:00:44.507282 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:00:44.507282 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:00:44.507282 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:00:44.507282 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 05:00:44.507282 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 05:00:44.507282 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 05:00:44.507282 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 05:00:44.507282 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 05:00:44.507282 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 05:00:44.507282 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 30 05:00:44.507282 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:00:44.507282 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 538/1089] 10.222.0.77 () {56 vars in 1120 bytes} [Tue Aug 30 05:00:44 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:00:44.512277 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24e34f02-701c-4232-879c-ea21045b63f7 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:44.536649 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77a75f3f-3ecf-4483-8500-b370bc1226ee None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:44.551561 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:44.573517 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3849642-db0c-4037-b2de-6be72fe4d24c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:44.574123 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3849642-db0c-4037-b2de-6be72fe4d24c None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:44.574394 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3849642-db0c-4037-b2de-6be72fe4d24c None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:44.574662 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3849642-db0c-4037-b2de-6be72fe4d24c None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:44.585019 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3849642-db0c-4037-b2de-6be72fe4d24c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 05:00:44.589693 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-e3849642-db0c-4037-b2de-6be72fe4d24c None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 05:00:44.589693 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:44.589693 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:00:44.589693 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:00:44.589693 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:00:44.589693 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:00:44.589693 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:00:44.589693 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:00:44.589693 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:00:44.589693 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:00:44.589693 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:00:44.589693 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:00:44.589693 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 05:00:44.589693 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 05:00:44.589693 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 05:00:44.589693 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 05:00:44.590637 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 05:00:44.590637 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 05:00:44.590637 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 05:00:44.590637 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 05:00:44.590637 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:00:44.590637 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:00:44.590637 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:00:44.590637 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:00:44.590637 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:00:44.590637 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:00:44.590637 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:00:44.590637 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 05:00:44.590637 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:00:44.590637 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:00:44.590637 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:00:44.590637 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:00:44.591666 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:00:44.591666 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:00:44.591666 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:00:44.591666 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:00:44.591666 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 05:00:44.591666 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 05:00:44.591666 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 05:00:44.591666 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 05:00:44.591666 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 05:00:44.591666 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 05:00:44.591666 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 05:00:44.591666 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:00:44.591666 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 552/1090] 10.222.0.77 () {56 vars in 1096 bytes} [Tue Aug 30 05:00:44 2022] GET /identity/v3/roles/admin => generated 83 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 05:00:44.596543 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3849642-db0c-4037-b2de-6be72fe4d24c None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:44.598290 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:44.611014 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24e34f02-701c-4232-879c-ea21045b63f7 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:44.646926 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dac73aba-f694-403a-8739-cf9d3853ac3f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:44.647523 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dac73aba-f694-403a-8739-cf9d3853ac3f None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:44.647800 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dac73aba-f694-403a-8739-cf9d3853ac3f None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:44.648071 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dac73aba-f694-403a-8739-cf9d3853ac3f None admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:44.648726 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dac73aba-f694-403a-8739-cf9d3853ac3f None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:44.649025 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dac73aba-f694-403a-8739-cf9d3853ac3f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:44.650391 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dac73aba-f694-403a-8739-cf9d3853ac3f None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:44.658075 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 539/1091] 10.222.0.77 () {56 vars in 1115 bytes} [Tue Aug 30 05:00:44 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 05:00:44.664065 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dac73aba-f694-403a-8739-cf9d3853ac3f None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:44.695944 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3849642-db0c-4037-b2de-6be72fe4d24c None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:44.729187 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-816291f3-9656-4214-9431-8737e61d2ba1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:44.729868 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-816291f3-9656-4214-9431-8737e61d2ba1 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:44.730213 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-816291f3-9656-4214-9431-8737e61d2ba1 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:44.730577 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-816291f3-9656-4214-9431-8737e61d2ba1 None admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:44.731348 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-816291f3-9656-4214-9431-8737e61d2ba1 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:44.731645 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-816291f3-9656-4214-9431-8737e61d2ba1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:44.732992 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-816291f3-9656-4214-9431-8737e61d2ba1 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:44.741890 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 553/1092] 10.222.0.77 () {56 vars in 1099 bytes} [Tue Aug 30 05:00:44 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 05:00:44.747585 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-816291f3-9656-4214-9431-8737e61d2ba1 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:44.762567 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dac73aba-f694-403a-8739-cf9d3853ac3f None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:44.779546 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:44.797195 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-479e718c-0feb-4b5c-bc05-e12b3231bbe7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:44.797819 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-479e718c-0feb-4b5c-bc05-e12b3231bbe7 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:44.798096 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-479e718c-0feb-4b5c-bc05-e12b3231bbe7 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:44.798365 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-479e718c-0feb-4b5c-bc05-e12b3231bbe7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:44.809756 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-479e718c-0feb-4b5c-bc05-e12b3231bbe7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:44.811249 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-479e718c-0feb-4b5c-bc05-e12b3231bbe7 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:44.825456 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-479e718c-0feb-4b5c-bc05-e12b3231bbe7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 05:00:44.825456 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:44.825456 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:00:44.825456 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:00:44.825456 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:00:44.825456 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:00:44.825456 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:00:44.825456 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:00:44.825456 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:00:44.825456 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:00:44.825456 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:00:44.825456 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:00:44.825456 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 05:00:44.825456 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 05:00:44.825456 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 05:00:44.825456 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 05:00:44.826609 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:00:44.826609 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:00:44.826609 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:00:44.826609 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:00:44.826609 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:00:44.826609 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:00:44.826609 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:00:44.826609 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 05:00:44.826609 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:00:44.826609 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:00:44.826609 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:00:44.826609 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:00:44.826609 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:00:44.826609 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:00:44.826609 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:00:44.826609 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:00:44.826609 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 05:00:44.827582 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 05:00:44.827582 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 05:00:44.827582 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:00:44.827582 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 540/1093] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 05:00:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 05:00:44.832323 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-479e718c-0feb-4b5c-bc05-e12b3231bbe7 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:44.834948 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:44.850719 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-816291f3-9656-4214-9431-8737e61d2ba1 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:44.883998 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-800e91e0-cdbb-4ace-a683-9005cc8d117e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:44.884550 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-800e91e0-cdbb-4ace-a683-9005cc8d117e None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:44.884825 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-800e91e0-cdbb-4ace-a683-9005cc8d117e None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:44.885093 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-800e91e0-cdbb-4ace-a683-9005cc8d117e None admin] PATH_INFO: `/v3/users/c4f1c3280e864ff1bc823775ea5ea3a5` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:44.886195 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-800e91e0-cdbb-4ace-a683-9005cc8d117e None admin] RBAC: Authorizing `identity:get_user(user_id=c4f1c3280e864ff1bc823775ea5ea3a5)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:44.887643 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-800e91e0-cdbb-4ace-a683-9005cc8d117e None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:44.889181 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 554/1094] 10.222.0.77 () {56 vars in 1177 bytes} [Tue Aug 30 05:00:44 2022] GET /identity/v3/users/c4f1c3280e864ff1bc823775ea5ea3a5 => generated 285 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 05:00:44.895011 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-800e91e0-cdbb-4ace-a683-9005cc8d117e None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:44.934904 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-479e718c-0feb-4b5c-bc05-e12b3231bbe7 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:44.967697 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02cbda20-7b35-42ab-9ba4-5249ee75b4d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:44.968294 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02cbda20-7b35-42ab-9ba4-5249ee75b4d7 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:44.968574 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02cbda20-7b35-42ab-9ba4-5249ee75b4d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:44.968846 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02cbda20-7b35-42ab-9ba4-5249ee75b4d7 None admin] PATH_INFO: `/v3/domains` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:44.969480 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02cbda20-7b35-42ab-9ba4-5249ee75b4d7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:44.969777 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02cbda20-7b35-42ab-9ba4-5249ee75b4d7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:44.971180 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02cbda20-7b35-42ab-9ba4-5249ee75b4d7 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:44.983435 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 541/1095] 10.222.0.77 () {56 vars in 1109 bytes} [Tue Aug 30 05:00:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 05:00:44.988947 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02cbda20-7b35-42ab-9ba4-5249ee75b4d7 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:44.993974 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-800e91e0-cdbb-4ace-a683-9005cc8d117e None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:44.995538 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:45.028976 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f462a7a-56ce-417e-9e69-ecabf1fc3c8b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:45.029570 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f462a7a-56ce-417e-9e69-ecabf1fc3c8b None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:45.029847 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f462a7a-56ce-417e-9e69-ecabf1fc3c8b None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:45.030116 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f462a7a-56ce-417e-9e69-ecabf1fc3c8b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:45.048257 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:45.086785 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02cbda20-7b35-42ab-9ba4-5249ee75b4d7 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:45.100511 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f462a7a-56ce-417e-9e69-ecabf1fc3c8b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:45.101948 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f462a7a-56ce-417e-9e69-ecabf1fc3c8b None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:45.120416 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-569b21fb-f509-4932-989d-ad892c2e160c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:45.121025 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-569b21fb-f509-4932-989d-ad892c2e160c None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:45.121405 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-569b21fb-f509-4932-989d-ad892c2e160c None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:45.121675 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-569b21fb-f509-4932-989d-ad892c2e160c None admin] PATH_INFO: `/v3/users/nova` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:45.128463 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-569b21fb-f509-4932-989d-ad892c2e160c None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:45.129879 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-569b21fb-f509-4932-989d-ad892c2e160c None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:45.141104 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-569b21fb-f509-4932-989d-ad892c2e160c None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 30 05:00:45.141104 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:45.141104 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:00:45.141104 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:00:45.141104 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:00:45.141104 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:00:45.141104 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:00:45.141104 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:00:45.141104 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:00:45.141104 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:00:45.141104 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:00:45.141104 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:00:45.141104 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 05:00:45.141104 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 05:00:45.141104 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 05:00:45.141104 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 05:00:45.142045 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:00:45.142045 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:00:45.142045 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 05:00:45.142045 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 05:00:45.142045 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 05:00:45.142045 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 05:00:45.142045 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:00:45.142045 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:00:45.142045 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:00:45.142045 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:00:45.142045 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:00:45.142045 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 05:00:45.142045 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:00:45.142045 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:00:45.142045 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:00:45.142045 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:00:45.142045 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:00:45.142045 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:00:45.143082 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:00:45.143082 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:00:45.143082 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 05:00:45.143082 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 05:00:45.143082 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 05:00:45.143082 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 05:00:45.143082 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 05:00:45.143082 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 05:00:45.143082 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 30 05:00:45.143082 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:00:45.143082 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 542/1096] 10.222.0.77 () {56 vars in 1093 bytes} [Tue Aug 30 05:00:44 2022] GET /identity/v3/users/nova => generated 82 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 05:00:45.148321 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-569b21fb-f509-4932-989d-ad892c2e160c None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:45.199133 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 555/1097] 10.222.0.77 () {58 vars in 1282 bytes} [Tue Aug 30 05:00:44 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 304 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Aug 30 05:00:45.205069 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f462a7a-56ce-417e-9e69-ecabf1fc3c8b None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:45.206876 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:45.249402 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-569b21fb-f509-4932-989d-ad892c2e160c None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:45.272435 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:45.283707 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2049bb77-d126-439d-8e3d-3a4274d9fdca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:45.284295 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2049bb77-d126-439d-8e3d-3a4274d9fdca None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:45.284569 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2049bb77-d126-439d-8e3d-3a4274d9fdca None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:45.284836 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2049bb77-d126-439d-8e3d-3a4274d9fdca None admin] PATH_INFO: `/v3/users` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:45.285550 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2049bb77-d126-439d-8e3d-3a4274d9fdca None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:45.285847 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2049bb77-d126-439d-8e3d-3a4274d9fdca None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:45.287822 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2049bb77-d126-439d-8e3d-3a4274d9fdca None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:45.304372 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 543/1098] 10.222.0.77 () {56 vars in 1133 bytes} [Tue Aug 30 05:00:45 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 05:00:45.304656 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f462a7a-56ce-417e-9e69-ecabf1fc3c8b None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:45.310073 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2049bb77-d126-439d-8e3d-3a4274d9fdca None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:45.339675 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13731803-c45f-4638-ad96-6601854a96e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:45.340298 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13731803-c45f-4638-ad96-6601854a96e3 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:45.340578 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13731803-c45f-4638-ad96-6601854a96e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:45.340847 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13731803-c45f-4638-ad96-6601854a96e3 None admin] PATH_INFO: `/v3/projects/c676bac7a20f42ce88c6f9805903d625` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:45.341952 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13731803-c45f-4638-ad96-6601854a96e3 None admin] RBAC: Authorizing `identity:get_project(project_id=c676bac7a20f42ce88c6f9805903d625)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:45.343425 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13731803-c45f-4638-ad96-6601854a96e3 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:45.345018 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 556/1099] 10.222.0.77 () {56 vars in 1186 bytes} [Tue Aug 30 05:00:45 2022] GET /identity/v3/projects/c676bac7a20f42ce88c6f9805903d625 => generated 311 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 05:00:45.351505 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13731803-c45f-4638-ad96-6601854a96e3 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:45.408896 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2049bb77-d126-439d-8e3d-3a4274d9fdca None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:45.420098 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:45.443074 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d0e00b0-c428-4a8b-a1f0-d192d67b560f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:45.444172 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d0e00b0-c428-4a8b-a1f0-d192d67b560f None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:45.444448 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d0e00b0-c428-4a8b-a1f0-d192d67b560f None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:45.444716 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d0e00b0-c428-4a8b-a1f0-d192d67b560f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:45.451518 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13731803-c45f-4638-ad96-6601854a96e3 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:45.486050 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4acf4184-afb3-47ff-9e31-9e925eaa6773 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:45.486647 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4acf4184-afb3-47ff-9e31-9e925eaa6773 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:45.486952 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4acf4184-afb3-47ff-9e31-9e925eaa6773 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:45.487223 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4acf4184-afb3-47ff-9e31-9e925eaa6773 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:45.487940 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4acf4184-afb3-47ff-9e31-9e925eaa6773 None admin] RBAC: Adding query filter params (role.id=9d9d5d2c270146488249b7ed943280db, scope.project.id=c676bac7a20f42ce88c6f9805903d625, user.id=c4f1c3280e864ff1bc823775ea5ea3a5) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:45.488244 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4acf4184-afb3-47ff-9e31-9e925eaa6773 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:45.489694 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4acf4184-afb3-47ff-9e31-9e925eaa6773 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:45.495615 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 557/1100] 10.222.0.77 () {56 vars in 1374 bytes} [Tue Aug 30 05:00:45 2022] GET /identity/v3/role_assignments?user.id=c4f1c3280e864ff1bc823775ea5ea3a5&scope.project.id=c676bac7a20f42ce88c6f9805903d625&role.id=9d9d5d2c270146488249b7ed943280db => generated 608 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 05:00:45.501269 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:45.523956 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d0e00b0-c428-4a8b-a1f0-d192d67b560f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:45.525442 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d0e00b0-c428-4a8b-a1f0-d192d67b560f None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:45.628423 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 544/1101] 10.222.0.77 () {58 vars in 1282 bytes} [Tue Aug 30 05:00:45 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 319 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Aug 30 05:00:45.634671 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4acf4184-afb3-47ff-9e31-9e925eaa6773 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:45.734738 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4acf4184-afb3-47ff-9e31-9e925eaa6773 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:45.767454 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5043be6b-4ab5-48b2-9f7a-b002a038af64 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:45.768067 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5043be6b-4ab5-48b2-9f7a-b002a038af64 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:45.768344 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5043be6b-4ab5-48b2-9f7a-b002a038af64 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:45.768612 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5043be6b-4ab5-48b2-9f7a-b002a038af64 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:45.770436 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:45.780629 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5043be6b-4ab5-48b2-9f7a-b002a038af64 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:45.782110 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5043be6b-4ab5-48b2-9f7a-b002a038af64 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:45.796019 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-5043be6b-4ab5-48b2-9f7a-b002a038af64 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 05:00:45.796019 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:45.796019 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:00:45.796019 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:00:45.796019 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:00:45.796019 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:00:45.796019 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:00:45.796019 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:00:45.796019 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:00:45.796019 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:00:45.796019 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:00:45.796019 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:00:45.796019 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 05:00:45.796019 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 05:00:45.796019 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 05:00:45.796019 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 05:00:45.796939 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:00:45.796939 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:00:45.796939 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:00:45.796939 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:00:45.796939 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:00:45.796939 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:00:45.796939 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:00:45.796939 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 05:00:45.796939 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:00:45.796939 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:00:45.796939 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:00:45.796939 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:00:45.796939 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:00:45.796939 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:00:45.796939 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:00:45.796939 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:00:45.796939 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 05:00:45.797918 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 05:00:45.797918 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 05:00:45.797918 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:00:45.797918 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 558/1102] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 05:00:45 2022] GET /identity/v3/domains/Default => generated 87 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:00:45.803490 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d0e00b0-c428-4a8b-a1f0-d192d67b560f None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:45.883528 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:45.902780 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d0e00b0-c428-4a8b-a1f0-d192d67b560f None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:45.935993 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2f45f48-fc83-458e-bb0b-badbfe4fc0c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:45.936588 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f45f48-fc83-458e-bb0b-badbfe4fc0c8 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:45.936869 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f45f48-fc83-458e-bb0b-badbfe4fc0c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:45.937141 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f45f48-fc83-458e-bb0b-badbfe4fc0c8 None admin] PATH_INFO: `/v3/domains` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:45.937771 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2f45f48-fc83-458e-bb0b-badbfe4fc0c8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:45.938071 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2f45f48-fc83-458e-bb0b-badbfe4fc0c8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:45.939463 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2f45f48-fc83-458e-bb0b-badbfe4fc0c8 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:45.951116 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 545/1103] 10.222.0.77 () {56 vars in 1109 bytes} [Tue Aug 30 05:00:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:45.956727 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5043be6b-4ab5-48b2-9f7a-b002a038af64 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:46.053898 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5043be6b-4ab5-48b2-9f7a-b002a038af64 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:46.086458 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b6ae61e-4791-4eb6-9e88-0f487e9a92bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:46.087088 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b6ae61e-4791-4eb6-9e88-0f487e9a92bb None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:46.087361 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b6ae61e-4791-4eb6-9e88-0f487e9a92bb None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:46.087622 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b6ae61e-4791-4eb6-9e88-0f487e9a92bb None admin] PATH_INFO: `/v3/projects/service` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:46.093925 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b6ae61e-4791-4eb6-9e88-0f487e9a92bb None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:46.095387 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b6ae61e-4791-4eb6-9e88-0f487e9a92bb None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:46.106437 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-7b6ae61e-4791-4eb6-9e88-0f487e9a92bb None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 05:00:46.106437 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:00:46.106437 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:00:46.106437 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:00:46.106437 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:00:46.106437 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:00:46.106437 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:00:46.106437 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:00:46.106437 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:00:46.106437 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:00:46.106437 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:00:46.106437 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:00:46.106437 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 05:00:46.106437 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 05:00:46.106437 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 05:00:46.107511 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 05:00:46.107511 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:00:46.107511 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:00:46.107511 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:00:46.107511 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:00:46.107511 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:00:46.107511 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:00:46.107511 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:00:46.107511 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 05:00:46.107511 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:00:46.107511 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:00:46.107511 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:00:46.107511 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:00:46.107511 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:00:46.107511 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:00:46.107511 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:00:46.107511 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:00:46.108792 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 05:00:46.108792 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 05:00:46.108792 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 05:00:46.108792 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 05:00:46.108792 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 05:00:46.108792 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 05:00:46.108792 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 05:00:46.108792 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:00:46.108792 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 559/1104] 10.222.0.77 () {56 vars in 1111 bytes} [Tue Aug 30 05:00:45 2022] GET /identity/v3/projects/service => generated 88 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 05:00:46.113332 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2f45f48-fc83-458e-bb0b-badbfe4fc0c8 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:46.209883 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2f45f48-fc83-458e-bb0b-badbfe4fc0c8 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:46.242608 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0598635-6ea3-4359-a11b-6cfc717eae49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:46.243229 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0598635-6ea3-4359-a11b-6cfc717eae49 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:46.243503 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0598635-6ea3-4359-a11b-6cfc717eae49 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:46.243769 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0598635-6ea3-4359-a11b-6cfc717eae49 None admin] PATH_INFO: `/v3/projects` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:46.244442 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0598635-6ea3-4359-a11b-6cfc717eae49 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:46.244743 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0598635-6ea3-4359-a11b-6cfc717eae49 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:46.246152 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0598635-6ea3-4359-a11b-6cfc717eae49 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:46.254730 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:46.258053 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 546/1105] 10.222.0.77 () {56 vars in 1148 bytes} [Tue Aug 30 05:00:46 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:46.264285 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b6ae61e-4791-4eb6-9e88-0f487e9a92bb None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:46.309547 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:46.360508 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b6ae61e-4791-4eb6-9e88-0f487e9a92bb None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:46.392682 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e5a627f-7319-40fe-94db-fce53be48f36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:46.393272 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e5a627f-7319-40fe-94db-fce53be48f36 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:46.393546 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e5a627f-7319-40fe-94db-fce53be48f36 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:46.393808 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e5a627f-7319-40fe-94db-fce53be48f36 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:46.394514 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e5a627f-7319-40fe-94db-fce53be48f36 None admin] RBAC: Adding query filter params (role.id=2d3c2095a28d4f4f8a8328c4b61b0015, scope.project.id=e9919141e3b947ab8d91b55bba4a7de6, user.id=886eb5e00bbf4ec292eb962b665b37d0) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:46.394835 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e5a627f-7319-40fe-94db-fce53be48f36 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:46.396243 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e5a627f-7319-40fe-94db-fce53be48f36 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:46.402008 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 560/1106] 10.222.0.77 () {56 vars in 1374 bytes} [Tue Aug 30 05:00:46 2022] GET /identity/v3/role_assignments?user.id=886eb5e00bbf4ec292eb962b665b37d0&scope.project.id=e9919141e3b947ab8d91b55bba4a7de6&role.id=2d3c2095a28d4f4f8a8328c4b61b0015 => generated 608 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:47.428895 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81d24123-8bfd-4bc6-8da9-2b4503f6edb1 None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:47.429424 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81d24123-8bfd-4bc6-8da9-2b4503f6edb1 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:47.429424 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81d24123-8bfd-4bc6-8da9-2b4503f6edb1 None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:47.430770 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 547/1107] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:00:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:00:47.435794 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a020779-82db-40b4-b2ee-9dce7f3ec6be None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:47.436236 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a020779-82db-40b4-b2ee-9dce7f3ec6be None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:47.436330 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a020779-82db-40b4-b2ee-9dce7f3ec6be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:47.454681 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-6a020779-82db-40b4-b2ee-9dce7f3ec6be None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:00:47.455966 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-6a020779-82db-40b4-b2ee-9dce7f3ec6be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:00:47.521167 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 561/1108] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:00:47 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 05:00:47.786869 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81d24123-8bfd-4bc6-8da9-2b4503f6edb1 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:47.884979 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81d24123-8bfd-4bc6-8da9-2b4503f6edb1 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:47.917277 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8acd7c4-f3eb-4e29-b676-1a03b2acced0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:47.917903 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8acd7c4-f3eb-4e29-b676-1a03b2acced0 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:47.918180 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8acd7c4-f3eb-4e29-b676-1a03b2acced0 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:47.918448 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8acd7c4-f3eb-4e29-b676-1a03b2acced0 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:47.919550 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8acd7c4-f3eb-4e29-b676-1a03b2acced0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:47.920998 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8acd7c4-f3eb-4e29-b676-1a03b2acced0 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:47.922518 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 548/1109] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 05:00:47 2022] GET /identity/v3/domains/default => generated 199 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:47.928557 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a020779-82db-40b4-b2ee-9dce7f3ec6be None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:47.967617 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:48.024710 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a020779-82db-40b4-b2ee-9dce7f3ec6be None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:48.058122 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f14742f5-53b1-4c81-8a0d-9a16f85fa5f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:48.058710 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f14742f5-53b1-4c81-8a0d-9a16f85fa5f2 None admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:48.059021 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f14742f5-53b1-4c81-8a0d-9a16f85fa5f2 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:48.059295 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f14742f5-53b1-4c81-8a0d-9a16f85fa5f2 None admin] PATH_INFO: `/v3/users` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:48.059936 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f14742f5-53b1-4c81-8a0d-9a16f85fa5f2 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:48.061350 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f14742f5-53b1-4c81-8a0d-9a16f85fa5f2 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:48.081320 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 562/1110] 10.222.0.77 () {60 vars in 1132 bytes} [Tue Aug 30 05:00:47 2022] POST /identity/v3/users => generated 286 bytes in 154 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:00:49.972551 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5272229a-f9ff-4644-a1c4-495ac011a8e1 None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:49.973079 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5272229a-f9ff-4644-a1c4-495ac011a8e1 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:49.973079 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5272229a-f9ff-4644-a1c4-495ac011a8e1 None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:49.974417 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 549/1111] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:00:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:00:49.980255 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca6fbf3-79bf-45bd-abe0-785da9a73c06 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:49.980622 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca6fbf3-79bf-45bd-abe0-785da9a73c06 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:49.980803 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca6fbf3-79bf-45bd-abe0-785da9a73c06 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:49.998354 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-5ca6fbf3-79bf-45bd-abe0-785da9a73c06 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:00:49.999559 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-5ca6fbf3-79bf-45bd-abe0-785da9a73c06 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:00:50.063505 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 563/1112] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:00:49 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 05:00:50.327831 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5272229a-f9ff-4644-a1c4-495ac011a8e1 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:50.393924 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:50.427098 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5272229a-f9ff-4644-a1c4-495ac011a8e1 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:50.460036 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc4c4d79-4c02-4b47-b499-89b8d96833a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:50.460679 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc4c4d79-4c02-4b47-b499-89b8d96833a3 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:50.460956 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc4c4d79-4c02-4b47-b499-89b8d96833a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:50.461226 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc4c4d79-4c02-4b47-b499-89b8d96833a3 None admin] PATH_INFO: `/v3/roles/3b9d5509a2f7478c9b4989d921daef2a` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:50.462617 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc4c4d79-4c02-4b47-b499-89b8d96833a3 None admin] RBAC: Authorizing `identity:get_role(role_id=3b9d5509a2f7478c9b4989d921daef2a)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:50.464011 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc4c4d79-4c02-4b47-b499-89b8d96833a3 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:50.465489 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 550/1113] 10.222.0.77 () {56 vars in 1177 bytes} [Tue Aug 30 05:00:50 2022] GET /identity/v3/roles/3b9d5509a2f7478c9b4989d921daef2a => generated 223 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:50.470991 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ca6fbf3-79bf-45bd-abe0-785da9a73c06 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:50.496872 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:50.567409 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ca6fbf3-79bf-45bd-abe0-785da9a73c06 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:50.600771 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3052931-cec0-4450-b983-4715be745852 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:50.601324 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3052931-cec0-4450-b983-4715be745852 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:50.601600 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3052931-cec0-4450-b983-4715be745852 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:50.601870 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3052931-cec0-4450-b983-4715be745852 None admin] PATH_INFO: `/v3/users/9f9c56564b6c4bd69de29f3e2ce8b575` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:50.614657 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3052931-cec0-4450-b983-4715be745852 None admin] RBAC: Authorizing `identity:get_user(user_id=9f9c56564b6c4bd69de29f3e2ce8b575)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:50.616115 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3052931-cec0-4450-b983-4715be745852 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:50.617680 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 564/1114] 10.222.0.77 () {56 vars in 1177 bytes} [Tue Aug 30 05:00:50 2022] GET /identity/v3/users/9f9c56564b6c4bd69de29f3e2ce8b575 => generated 286 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:50.623275 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc4c4d79-4c02-4b47-b499-89b8d96833a3 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:50.719446 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc4c4d79-4c02-4b47-b499-89b8d96833a3 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:50.752201 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7712dcd1-d944-40ea-8f76-9a73da8afd6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:50.752783 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7712dcd1-d944-40ea-8f76-9a73da8afd6a None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:50.753057 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7712dcd1-d944-40ea-8f76-9a73da8afd6a None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:50.753323 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7712dcd1-d944-40ea-8f76-9a73da8afd6a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:50.757091 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:50.823623 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7712dcd1-d944-40ea-8f76-9a73da8afd6a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:50.825100 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7712dcd1-d944-40ea-8f76-9a73da8afd6a None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:50.921101 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 551/1115] 10.222.0.77 () {58 vars in 1282 bytes} [Tue Aug 30 05:00:50 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 298 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 30 05:00:50.926948 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3052931-cec0-4450-b983-4715be745852 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:51.008635 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:51.024246 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3052931-cec0-4450-b983-4715be745852 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:51.056749 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2da2894-ab65-4a62-aa76-45b27697ebd6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:51.057363 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2da2894-ab65-4a62-aa76-45b27697ebd6 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:51.057642 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2da2894-ab65-4a62-aa76-45b27697ebd6 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:51.057910 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2da2894-ab65-4a62-aa76-45b27697ebd6 None admin] PATH_INFO: `/v3/projects/c676bac7a20f42ce88c6f9805903d625` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:51.059019 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2da2894-ab65-4a62-aa76-45b27697ebd6 None admin] RBAC: Authorizing `identity:get_project(project_id=c676bac7a20f42ce88c6f9805903d625)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:51.060434 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2da2894-ab65-4a62-aa76-45b27697ebd6 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:51.062012 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 565/1116] 10.222.0.77 () {56 vars in 1186 bytes} [Tue Aug 30 05:00:50 2022] GET /identity/v3/projects/c676bac7a20f42ce88c6f9805903d625 => generated 311 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:51.068043 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7712dcd1-d944-40ea-8f76-9a73da8afd6a None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:51.101464 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:51.163758 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7712dcd1-d944-40ea-8f76-9a73da8afd6a None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:51.196409 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9f558ee-7e6f-4ccf-920e-da1a542a3ea7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:51.196933 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f558ee-7e6f-4ccf-920e-da1a542a3ea7 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:51.197208 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f558ee-7e6f-4ccf-920e-da1a542a3ea7 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:51.197475 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f558ee-7e6f-4ccf-920e-da1a542a3ea7 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:51.198176 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9f558ee-7e6f-4ccf-920e-da1a542a3ea7 None admin] RBAC: Adding query filter params (role.id=3b9d5509a2f7478c9b4989d921daef2a, scope.project.id=c676bac7a20f42ce88c6f9805903d625, user.id=9f9c56564b6c4bd69de29f3e2ce8b575) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:51.198477 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9f558ee-7e6f-4ccf-920e-da1a542a3ea7 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:51.199907 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9f558ee-7e6f-4ccf-920e-da1a542a3ea7 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:51.205097 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 552/1117] 10.222.0.77 () {56 vars in 1374 bytes} [Tue Aug 30 05:00:51 2022] GET /identity/v3/role_assignments?user.id=9f9c56564b6c4bd69de29f3e2ce8b575&scope.project.id=c676bac7a20f42ce88c6f9805903d625&role.id=3b9d5509a2f7478c9b4989d921daef2a => generated 259 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:53.051136 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-191301fe-5237-44bf-87c5-1665c2a2067a None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:53.051719 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-191301fe-5237-44bf-87c5-1665c2a2067a None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:53.051719 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-191301fe-5237-44bf-87c5-1665c2a2067a None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:53.053522 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 566/1118] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:00:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:00:53.058699 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea1acca3-aab5-4c79-8f67-8e57dd107ae8 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:53.059123 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea1acca3-aab5-4c79-8f67-8e57dd107ae8 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:53.059291 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea1acca3-aab5-4c79-8f67-8e57dd107ae8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:53.077617 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-ea1acca3-aab5-4c79-8f67-8e57dd107ae8 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:00:53.078808 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-ea1acca3-aab5-4c79-8f67-8e57dd107ae8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:00:53.143204 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 553/1119] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:00:53 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 05:00:53.407957 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-191301fe-5237-44bf-87c5-1665c2a2067a None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:53.509434 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-191301fe-5237-44bf-87c5-1665c2a2067a None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:53.542039 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c56121c-f9fc-4cbe-9d13-6dbfee08f1ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:53.542649 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c56121c-f9fc-4cbe-9d13-6dbfee08f1ab None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:53.542956 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c56121c-f9fc-4cbe-9d13-6dbfee08f1ab None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:53.543227 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c56121c-f9fc-4cbe-9d13-6dbfee08f1ab None admin] PATH_INFO: `/v3/roles/3b9d5509a2f7478c9b4989d921daef2a` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:53.544593 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c56121c-f9fc-4cbe-9d13-6dbfee08f1ab None admin] RBAC: Authorizing `identity:get_role(role_id=3b9d5509a2f7478c9b4989d921daef2a)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:53.545945 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c56121c-f9fc-4cbe-9d13-6dbfee08f1ab None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:53.547270 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 567/1120] 10.222.0.77 () {56 vars in 1177 bytes} [Tue Aug 30 05:00:53 2022] GET /identity/v3/roles/3b9d5509a2f7478c9b4989d921daef2a => generated 223 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 05:00:53.553541 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea1acca3-aab5-4c79-8f67-8e57dd107ae8 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:53.582209 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:53.650245 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea1acca3-aab5-4c79-8f67-8e57dd107ae8 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:53.684197 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44792e4c-c1d6-4f6d-bd71-b96ac463ab4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:53.684749 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44792e4c-c1d6-4f6d-bd71-b96ac463ab4b None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:53.685028 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44792e4c-c1d6-4f6d-bd71-b96ac463ab4b None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:53.685298 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44792e4c-c1d6-4f6d-bd71-b96ac463ab4b None admin] PATH_INFO: `/v3/users/9f9c56564b6c4bd69de29f3e2ce8b575` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:53.686399 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44792e4c-c1d6-4f6d-bd71-b96ac463ab4b None admin] RBAC: Authorizing `identity:get_user(user_id=9f9c56564b6c4bd69de29f3e2ce8b575)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:53.687849 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44792e4c-c1d6-4f6d-bd71-b96ac463ab4b None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:53.689398 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 554/1121] 10.222.0.77 () {56 vars in 1177 bytes} [Tue Aug 30 05:00:53 2022] GET /identity/v3/users/9f9c56564b6c4bd69de29f3e2ce8b575 => generated 286 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:53.694947 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c56121c-f9fc-4cbe-9d13-6dbfee08f1ab None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:53.716505 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:53.792727 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c56121c-f9fc-4cbe-9d13-6dbfee08f1ab None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:53.824618 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c71bfc07-393a-4072-aa9e-9ac9098e5814 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:53.825221 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c71bfc07-393a-4072-aa9e-9ac9098e5814 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:53.825496 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c71bfc07-393a-4072-aa9e-9ac9098e5814 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:53.825764 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c71bfc07-393a-4072-aa9e-9ac9098e5814 None admin] PATH_INFO: `/v3/projects/c676bac7a20f42ce88c6f9805903d625` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:53.826866 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c71bfc07-393a-4072-aa9e-9ac9098e5814 None admin] RBAC: Authorizing `identity:get_project(project_id=c676bac7a20f42ce88c6f9805903d625)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:53.828275 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c71bfc07-393a-4072-aa9e-9ac9098e5814 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:53.829849 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 568/1122] 10.222.0.77 () {56 vars in 1186 bytes} [Tue Aug 30 05:00:53 2022] GET /identity/v3/projects/c676bac7a20f42ce88c6f9805903d625 => generated 311 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:53.836070 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44792e4c-c1d6-4f6d-bd71-b96ac463ab4b None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:53.928776 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:53.932929 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44792e4c-c1d6-4f6d-bd71-b96ac463ab4b None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:53.965704 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8585df7-bfbb-4e84-aaac-c1bcc4533272 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:53.966276 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8585df7-bfbb-4e84-aaac-c1bcc4533272 None admin] REQUEST_METHOD: `PUT` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:53.966553 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8585df7-bfbb-4e84-aaac-c1bcc4533272 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:53.966845 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8585df7-bfbb-4e84-aaac-c1bcc4533272 None admin] PATH_INFO: `/v3/projects/c676bac7a20f42ce88c6f9805903d625/users/9f9c56564b6c4bd69de29f3e2ce8b575/roles/3b9d5509a2f7478c9b4989d921daef2a` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:53.968928 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8585df7-bfbb-4e84-aaac-c1bcc4533272 None admin] RBAC: Authorizing `identity:create_grant(project_id=c676bac7a20f42ce88c6f9805903d625, user_id=9f9c56564b6c4bd69de29f3e2ce8b575, role_id=3b9d5509a2f7478c9b4989d921daef2a)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:53.970343 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8585df7-bfbb-4e84-aaac-c1bcc4533272 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:53.982880 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 555/1123] 10.222.0.77 () {58 vars in 1439 bytes} [Tue Aug 30 05:00:53 2022] PUT /identity/v3/projects/c676bac7a20f42ce88c6f9805903d625/users/9f9c56564b6c4bd69de29f3e2ce8b575/roles/3b9d5509a2f7478c9b4989d921daef2a => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:00:55.844978 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6db53aa-9d63-486e-8026-088b5a9e4825 None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:55.845496 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6db53aa-9d63-486e-8026-088b5a9e4825 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:55.845496 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6db53aa-9d63-486e-8026-088b5a9e4825 None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:55.846862 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 569/1124] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:00:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:00:55.852453 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0e4ef5a-7429-48a8-a4c7-9bfa409c4874 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:55.853131 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0e4ef5a-7429-48a8-a4c7-9bfa409c4874 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:55.853261 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0e4ef5a-7429-48a8-a4c7-9bfa409c4874 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:55.871424 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-f0e4ef5a-7429-48a8-a4c7-9bfa409c4874 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:00:55.872584 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-f0e4ef5a-7429-48a8-a4c7-9bfa409c4874 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:00:55.937518 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 556/1125] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:00:55 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 05:00:56.207189 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6db53aa-9d63-486e-8026-088b5a9e4825 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:56.292414 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:56.305721 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6db53aa-9d63-486e-8026-088b5a9e4825 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:56.338645 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ded03e8-c1b4-450a-947f-b679193918b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:56.339279 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ded03e8-c1b4-450a-947f-b679193918b7 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:56.339553 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ded03e8-c1b4-450a-947f-b679193918b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:56.339818 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ded03e8-c1b4-450a-947f-b679193918b7 None admin] PATH_INFO: `/v3/roles/3b9d5509a2f7478c9b4989d921daef2a` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:56.341183 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ded03e8-c1b4-450a-947f-b679193918b7 None admin] RBAC: Authorizing `identity:get_role(role_id=3b9d5509a2f7478c9b4989d921daef2a)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:56.342538 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ded03e8-c1b4-450a-947f-b679193918b7 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:56.343841 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 570/1126] 10.222.0.77 () {56 vars in 1177 bytes} [Tue Aug 30 05:00:56 2022] GET /identity/v3/roles/3b9d5509a2f7478c9b4989d921daef2a => generated 223 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:56.349444 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0e4ef5a-7429-48a8-a4c7-9bfa409c4874 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:56.432788 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:56.444952 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0e4ef5a-7429-48a8-a4c7-9bfa409c4874 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:56.477781 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-193bd355-cf77-4c8a-88e5-b1ed099957fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:56.478401 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-193bd355-cf77-4c8a-88e5-b1ed099957fe None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:56.478673 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-193bd355-cf77-4c8a-88e5-b1ed099957fe None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:56.478972 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-193bd355-cf77-4c8a-88e5-b1ed099957fe None admin] PATH_INFO: `/v3/users/9f9c56564b6c4bd69de29f3e2ce8b575` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:56.480059 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-193bd355-cf77-4c8a-88e5-b1ed099957fe None admin] RBAC: Authorizing `identity:get_user(user_id=9f9c56564b6c4bd69de29f3e2ce8b575)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:56.481479 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-193bd355-cf77-4c8a-88e5-b1ed099957fe None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:56.483057 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 557/1127] 10.222.0.77 () {56 vars in 1177 bytes} [Tue Aug 30 05:00:56 2022] GET /identity/v3/users/9f9c56564b6c4bd69de29f3e2ce8b575 => generated 286 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:56.488741 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ded03e8-c1b4-450a-947f-b679193918b7 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:56.584206 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ded03e8-c1b4-450a-947f-b679193918b7 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:56.617563 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-336329e5-0cb7-4ea0-82b8-0387e21ba337 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:56.618144 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-336329e5-0cb7-4ea0-82b8-0387e21ba337 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:56.618420 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-336329e5-0cb7-4ea0-82b8-0387e21ba337 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:56.618686 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-336329e5-0cb7-4ea0-82b8-0387e21ba337 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:56.641944 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:56.684070 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-336329e5-0cb7-4ea0-82b8-0387e21ba337 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:56.685495 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-336329e5-0cb7-4ea0-82b8-0387e21ba337 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:56.779721 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 571/1128] 10.222.0.77 () {58 vars in 1282 bytes} [Tue Aug 30 05:00:56 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 291 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 30 05:00:56.785471 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-193bd355-cf77-4c8a-88e5-b1ed099957fe None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:56.881239 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-193bd355-cf77-4c8a-88e5-b1ed099957fe None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:56.914687 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e41b89a-40fe-4f2f-8340-c5fffde01d0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:56.915314 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e41b89a-40fe-4f2f-8340-c5fffde01d0d None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:56.915591 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e41b89a-40fe-4f2f-8340-c5fffde01d0d None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:56.915861 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e41b89a-40fe-4f2f-8340-c5fffde01d0d None admin] PATH_INFO: `/v3/projects/c676bac7a20f42ce88c6f9805903d625` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:56.916948 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e41b89a-40fe-4f2f-8340-c5fffde01d0d None admin] RBAC: Authorizing `identity:get_project(project_id=c676bac7a20f42ce88c6f9805903d625)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:56.918360 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e41b89a-40fe-4f2f-8340-c5fffde01d0d None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:56.919980 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 558/1129] 10.222.0.77 () {56 vars in 1186 bytes} [Tue Aug 30 05:00:56 2022] GET /identity/v3/projects/c676bac7a20f42ce88c6f9805903d625 => generated 311 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:56.926065 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-336329e5-0cb7-4ea0-82b8-0387e21ba337 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:56.980692 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:57.023384 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-336329e5-0cb7-4ea0-82b8-0387e21ba337 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:57.056699 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a45fdd59-2893-4771-a0b1-82f1b339eb51 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:57.057282 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a45fdd59-2893-4771-a0b1-82f1b339eb51 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:57.057564 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a45fdd59-2893-4771-a0b1-82f1b339eb51 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:57.057838 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a45fdd59-2893-4771-a0b1-82f1b339eb51 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:57.058551 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a45fdd59-2893-4771-a0b1-82f1b339eb51 None admin] RBAC: Adding query filter params (role.id=3b9d5509a2f7478c9b4989d921daef2a, scope.project.id=c676bac7a20f42ce88c6f9805903d625, user.id=9f9c56564b6c4bd69de29f3e2ce8b575) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:00:57.058891 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a45fdd59-2893-4771-a0b1-82f1b339eb51 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:57.060310 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a45fdd59-2893-4771-a0b1-82f1b339eb51 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:57.065894 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 572/1130] 10.222.0.77 () {56 vars in 1374 bytes} [Tue Aug 30 05:00:56 2022] GET /identity/v3/role_assignments?user.id=9f9c56564b6c4bd69de29f3e2ce8b575&scope.project.id=c676bac7a20f42ce88c6f9805903d625&role.id=3b9d5509a2f7478c9b4989d921daef2a => generated 608 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:58.953443 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0782e623-3d6b-45e2-8b37-144aedeec5a6 None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:58.953998 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0782e623-3d6b-45e2-8b37-144aedeec5a6 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:58.953998 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0782e623-3d6b-45e2-8b37-144aedeec5a6 None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:58.955365 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 559/1131] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:00:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:00:58.960538 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c23fbe9-54c4-44d6-a9fb-bf9cf2171e00 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:58.960956 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c23fbe9-54c4-44d6-a9fb-bf9cf2171e00 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:58.961060 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c23fbe9-54c4-44d6-a9fb-bf9cf2171e00 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:58.978898 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-7c23fbe9-54c4-44d6-a9fb-bf9cf2171e00 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:00:58.980247 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-7c23fbe9-54c4-44d6-a9fb-bf9cf2171e00 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:00:59.043847 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 573/1132] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:00:58 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 05:00:59.310713 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0782e623-3d6b-45e2-8b37-144aedeec5a6 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:59.335056 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:59.408238 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0782e623-3d6b-45e2-8b37-144aedeec5a6 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:59.441225 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-664f2b55-0fb8-4546-a731-f55c397b35f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:59.441828 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-664f2b55-0fb8-4546-a731-f55c397b35f4 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:59.442104 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-664f2b55-0fb8-4546-a731-f55c397b35f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:59.442373 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-664f2b55-0fb8-4546-a731-f55c397b35f4 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:59.443470 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-664f2b55-0fb8-4546-a731-f55c397b35f4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:59.444916 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-664f2b55-0fb8-4546-a731-f55c397b35f4 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:59.446439 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 560/1133] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 05:00:59 2022] GET /identity/v3/domains/default => generated 199 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:00:59.452067 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c23fbe9-54c4-44d6-a9fb-bf9cf2171e00 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:00:59.500481 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:00:59.548313 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c23fbe9-54c4-44d6-a9fb-bf9cf2171e00 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:00:59.580620 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a928883-01c7-4716-b48b-56bc12cc0ade None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:00:59.581151 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a928883-01c7-4716-b48b-56bc12cc0ade None admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:00:59.581423 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a928883-01c7-4716-b48b-56bc12cc0ade None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:00:59.581687 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a928883-01c7-4716-b48b-56bc12cc0ade None admin] PATH_INFO: `/v3/projects` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:00:59.582314 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a928883-01c7-4716-b48b-56bc12cc0ade None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:00:59.583744 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a928883-01c7-4716-b48b-56bc12cc0ade None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:00:59.609425 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 574/1134] 10.222.0.77 () {60 vars in 1141 bytes} [Tue Aug 30 05:00:59 2022] POST /identity/v3/projects => generated 311 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:01:01.508439 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8221ffc-7002-4404-bc9e-90aa62484e33 None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:01:01.508898 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8221ffc-7002-4404-bc9e-90aa62484e33 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:01:01.508989 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8221ffc-7002-4404-bc9e-90aa62484e33 None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:01:01.510315 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 561/1135] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:01:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 05:01:01.515499 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-549cf643-f424-4f12-8f33-9565c564e6af None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:01:01.515914 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-549cf643-f424-4f12-8f33-9565c564e6af None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:01:01.516046 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-549cf643-f424-4f12-8f33-9565c564e6af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:01:01.533736 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-549cf643-f424-4f12-8f33-9565c564e6af None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:01:01.534932 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-549cf643-f424-4f12-8f33-9565c564e6af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:01:01.598433 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 575/1136] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:01:01 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 05:01:01.867432 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8221ffc-7002-4404-bc9e-90aa62484e33 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:01:01.957305 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:01:01.967397 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8221ffc-7002-4404-bc9e-90aa62484e33 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:01:02.000894 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4d4db9f-fa29-446a-85e0-c3dbcf56f819 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:01:02.001497 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d4db9f-fa29-446a-85e0-c3dbcf56f819 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:01:02.001773 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d4db9f-fa29-446a-85e0-c3dbcf56f819 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:01:02.002041 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d4db9f-fa29-446a-85e0-c3dbcf56f819 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:01:02.003151 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4d4db9f-fa29-446a-85e0-c3dbcf56f819 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:01:02.004571 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4d4db9f-fa29-446a-85e0-c3dbcf56f819 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:01:02.006108 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 562/1137] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 05:01:01 2022] GET /identity/v3/domains/default => generated 199 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:01:02.011766 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-549cf643-f424-4f12-8f33-9565c564e6af None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:01:02.041633 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:01:02.107497 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-549cf643-f424-4f12-8f33-9565c564e6af None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:01:02.139177 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-065c90a7-af3a-4975-9be8-03bfdb8c77fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:01:02.139707 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-065c90a7-af3a-4975-9be8-03bfdb8c77fc None admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:01:02.139980 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-065c90a7-af3a-4975-9be8-03bfdb8c77fc None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:01:02.140245 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-065c90a7-af3a-4975-9be8-03bfdb8c77fc None admin] PATH_INFO: `/v3/users` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:01:02.140883 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-065c90a7-af3a-4975-9be8-03bfdb8c77fc None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:01:02.142287 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-065c90a7-af3a-4975-9be8-03bfdb8c77fc None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:01:02.163642 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 576/1138] 10.222.0.77 () {60 vars in 1132 bytes} [Tue Aug 30 05:01:02 2022] POST /identity/v3/users => generated 286 bytes in 152 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:01:04.058025 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f7ad2f8-f4b2-422a-8e15-6f871ce7f6f9 None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:01:04.058576 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f7ad2f8-f4b2-422a-8e15-6f871ce7f6f9 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:01:04.058576 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f7ad2f8-f4b2-422a-8e15-6f871ce7f6f9 None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:01:04.059939 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 563/1139] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:01:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:01:04.065169 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45fe842a-d403-4f97-94a9-02f844487a2b None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:01:04.065574 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45fe842a-d403-4f97-94a9-02f844487a2b None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:01:04.065716 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45fe842a-d403-4f97-94a9-02f844487a2b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:01:04.083950 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-45fe842a-d403-4f97-94a9-02f844487a2b None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:01:04.085810 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-45fe842a-d403-4f97-94a9-02f844487a2b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:01:04.150418 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 577/1140] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:01:04 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 05:01:04.416863 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f7ad2f8-f4b2-422a-8e15-6f871ce7f6f9 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:01:04.515907 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f7ad2f8-f4b2-422a-8e15-6f871ce7f6f9 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:01:04.548222 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-934e056b-a158-43ed-a5c8-7a5f5b162947 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:01:04.548831 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-934e056b-a158-43ed-a5c8-7a5f5b162947 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:01:04.549107 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-934e056b-a158-43ed-a5c8-7a5f5b162947 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:01:04.549374 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-934e056b-a158-43ed-a5c8-7a5f5b162947 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:01:04.559638 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-934e056b-a158-43ed-a5c8-7a5f5b162947 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 05:01:04.564357 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-934e056b-a158-43ed-a5c8-7a5f5b162947 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 05:01:04.564357 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:01:04.564357 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:01:04.564357 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:01:04.564357 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:01:04.564357 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:01:04.564357 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:01:04.564357 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:01:04.564357 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:01:04.564357 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:01:04.564357 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:01:04.564357 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:01:04.564357 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 05:01:04.564357 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 05:01:04.564357 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 05:01:04.564357 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 05:01:04.565525 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 05:01:04.565525 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 05:01:04.565525 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 05:01:04.565525 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 05:01:04.565525 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:01:04.565525 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:01:04.565525 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:01:04.565525 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:01:04.565525 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:01:04.565525 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:01:04.565525 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:01:04.565525 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 05:01:04.565525 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:01:04.565525 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:01:04.565525 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:01:04.565525 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:01:04.566468 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:01:04.566468 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:01:04.566468 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:01:04.566468 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:01:04.566468 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 05:01:04.566468 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 05:01:04.566468 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 05:01:04.566468 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 05:01:04.566468 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 05:01:04.566468 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 05:01:04.566468 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 05:01:04.566468 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:01:04.566468 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 564/1141] 10.222.0.77 () {56 vars in 1096 bytes} [Tue Aug 30 05:01:04 2022] GET /identity/v3/roles/admin => generated 83 bytes in 149 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:01:04.571394 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45fe842a-d403-4f97-94a9-02f844487a2b None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:01:04.588891 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:01:04.667011 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45fe842a-d403-4f97-94a9-02f844487a2b None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:01:04.699527 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbe74bd5-503f-4285-9d68-57c626f42453 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:01:04.700113 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe74bd5-503f-4285-9d68-57c626f42453 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:01:04.700390 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe74bd5-503f-4285-9d68-57c626f42453 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:01:04.700660 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe74bd5-503f-4285-9d68-57c626f42453 None admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:01:04.701306 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbe74bd5-503f-4285-9d68-57c626f42453 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:01:04.701601 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbe74bd5-503f-4285-9d68-57c626f42453 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:01:04.702978 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbe74bd5-503f-4285-9d68-57c626f42453 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:01:04.710642 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 578/1142] 10.222.0.77 () {56 vars in 1099 bytes} [Tue Aug 30 05:01:04 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:01:04.716204 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-934e056b-a158-43ed-a5c8-7a5f5b162947 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:01:04.738658 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:01:04.814604 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-934e056b-a158-43ed-a5c8-7a5f5b162947 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:01:04.846828 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f6877fb-bda3-42af-b53a-228971e13a82 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:01:04.847455 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f6877fb-bda3-42af-b53a-228971e13a82 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:01:04.847731 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f6877fb-bda3-42af-b53a-228971e13a82 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:01:04.848002 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f6877fb-bda3-42af-b53a-228971e13a82 None admin] PATH_INFO: `/v3/users/2aa696d004424700ac807ab755d8a1c6` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:01:04.861342 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f6877fb-bda3-42af-b53a-228971e13a82 None admin] RBAC: Authorizing `identity:get_user(user_id=2aa696d004424700ac807ab755d8a1c6)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:01:04.863349 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f6877fb-bda3-42af-b53a-228971e13a82 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:01:04.864945 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 565/1143] 10.222.0.77 () {56 vars in 1177 bytes} [Tue Aug 30 05:01:04 2022] GET /identity/v3/users/2aa696d004424700ac807ab755d8a1c6 => generated 286 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:01:04.871409 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbe74bd5-503f-4285-9d68-57c626f42453 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:01:04.951455 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:01:04.970659 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbe74bd5-503f-4285-9d68-57c626f42453 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:01:05.002918 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72c10a5e-5cd5-493c-8bbb-bb6d3790caba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:01:05.003498 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72c10a5e-5cd5-493c-8bbb-bb6d3790caba None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:01:05.003770 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72c10a5e-5cd5-493c-8bbb-bb6d3790caba None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:01:05.004034 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72c10a5e-5cd5-493c-8bbb-bb6d3790caba None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:01:05.068524 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72c10a5e-5cd5-493c-8bbb-bb6d3790caba None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:01:05.069970 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72c10a5e-5cd5-493c-8bbb-bb6d3790caba None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:01:05.152210 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:01:05.173038 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 579/1144] 10.222.0.77 () {58 vars in 1282 bytes} [Tue Aug 30 05:01:04 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 302 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 30 05:01:05.179060 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f6877fb-bda3-42af-b53a-228971e13a82 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:01:05.253143 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:01:05.274116 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f6877fb-bda3-42af-b53a-228971e13a82 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:01:05.306702 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c6bee6d-a38d-486e-a534-797468fedea5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:01:05.307337 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c6bee6d-a38d-486e-a534-797468fedea5 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:01:05.307615 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c6bee6d-a38d-486e-a534-797468fedea5 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:01:05.307885 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c6bee6d-a38d-486e-a534-797468fedea5 None admin] PATH_INFO: `/v3/projects/322586836c984b49b28c8c731d860632` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:01:05.308969 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c6bee6d-a38d-486e-a534-797468fedea5 None admin] RBAC: Authorizing `identity:get_project(project_id=322586836c984b49b28c8c731d860632)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:01:05.310384 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c6bee6d-a38d-486e-a534-797468fedea5 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:01:05.312012 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 566/1145] 10.222.0.77 () {56 vars in 1186 bytes} [Tue Aug 30 05:01:05 2022] GET /identity/v3/projects/322586836c984b49b28c8c731d860632 => generated 311 bytes in 133 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:01:05.317909 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72c10a5e-5cd5-493c-8bbb-bb6d3790caba None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:01:05.413975 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72c10a5e-5cd5-493c-8bbb-bb6d3790caba None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:01:05.447002 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee58cbbc-c4fd-49b1-a53f-c3beea8e5f9b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:01:05.447580 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee58cbbc-c4fd-49b1-a53f-c3beea8e5f9b None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:01:05.447856 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee58cbbc-c4fd-49b1-a53f-c3beea8e5f9b None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:01:05.448126 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee58cbbc-c4fd-49b1-a53f-c3beea8e5f9b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:01:05.448833 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee58cbbc-c4fd-49b1-a53f-c3beea8e5f9b None admin] RBAC: Adding query filter params (role.id=9d9d5d2c270146488249b7ed943280db, scope.project.id=322586836c984b49b28c8c731d860632, user.id=2aa696d004424700ac807ab755d8a1c6) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:01:05.449136 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee58cbbc-c4fd-49b1-a53f-c3beea8e5f9b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:01:05.450542 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee58cbbc-c4fd-49b1-a53f-c3beea8e5f9b None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:01:05.455785 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 580/1146] 10.222.0.77 () {56 vars in 1374 bytes} [Tue Aug 30 05:01:05 2022] GET /identity/v3/role_assignments?user.id=2aa696d004424700ac807ab755d8a1c6&scope.project.id=322586836c984b49b28c8c731d860632&role.id=9d9d5d2c270146488249b7ed943280db => generated 259 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:01:07.289066 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dac34a1-ee49-45c0-8355-c88552477b47 None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:01:07.289625 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dac34a1-ee49-45c0-8355-c88552477b47 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:01:07.289625 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dac34a1-ee49-45c0-8355-c88552477b47 None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:01:07.291002 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 567/1147] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:01:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:01:07.296261 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea284451-cbde-45a9-8ad2-81cee0e092fe None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:01:07.296650 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea284451-cbde-45a9-8ad2-81cee0e092fe None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:01:07.296808 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea284451-cbde-45a9-8ad2-81cee0e092fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:01:07.315045 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-ea284451-cbde-45a9-8ad2-81cee0e092fe None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:01:07.316218 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-ea284451-cbde-45a9-8ad2-81cee0e092fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:01:07.354780 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:01:07.382177 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 581/1148] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:01:07 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 05:01:07.649765 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dac34a1-ee49-45c0-8355-c88552477b47 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:01:07.748676 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dac34a1-ee49-45c0-8355-c88552477b47 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:01:07.780922 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-413612f9-bd6c-4e2e-902d-0d4b55c36b47 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:01:07.781529 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-413612f9-bd6c-4e2e-902d-0d4b55c36b47 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:01:07.781806 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-413612f9-bd6c-4e2e-902d-0d4b55c36b47 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:01:07.782074 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-413612f9-bd6c-4e2e-902d-0d4b55c36b47 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:01:07.792340 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-413612f9-bd6c-4e2e-902d-0d4b55c36b47 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 05:01:07.797097 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-413612f9-bd6c-4e2e-902d-0d4b55c36b47 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 05:01:07.797097 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:01:07.797097 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:01:07.797097 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:01:07.797097 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:01:07.797097 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:01:07.797097 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:01:07.797097 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:01:07.797097 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:01:07.797097 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:01:07.797097 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:01:07.797097 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:01:07.797097 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 05:01:07.797097 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 05:01:07.797097 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 05:01:07.797097 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 05:01:07.798187 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 05:01:07.798187 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 05:01:07.798187 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 05:01:07.798187 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 05:01:07.798187 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:01:07.798187 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:01:07.798187 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:01:07.798187 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:01:07.798187 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:01:07.798187 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:01:07.798187 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:01:07.798187 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 05:01:07.798187 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:01:07.798187 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:01:07.798187 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:01:07.798187 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:01:07.799228 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:01:07.799228 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:01:07.799228 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:01:07.799228 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:01:07.799228 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 05:01:07.799228 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 05:01:07.799228 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 05:01:07.799228 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 05:01:07.799228 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 05:01:07.799228 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 05:01:07.799228 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 05:01:07.799228 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:01:07.799228 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 568/1149] 10.222.0.77 () {56 vars in 1096 bytes} [Tue Aug 30 05:01:07 2022] GET /identity/v3/roles/admin => generated 83 bytes in 149 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:01:07.804540 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea284451-cbde-45a9-8ad2-81cee0e092fe None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:01:07.899466 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea284451-cbde-45a9-8ad2-81cee0e092fe None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:01:07.933480 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4c17cc4-870d-4256-90a3-64f1457bd72d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:01:07.934015 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c17cc4-870d-4256-90a3-64f1457bd72d None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:01:07.934295 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c17cc4-870d-4256-90a3-64f1457bd72d None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:01:07.934563 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c17cc4-870d-4256-90a3-64f1457bd72d None admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:01:07.935239 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4c17cc4-870d-4256-90a3-64f1457bd72d None admin] RBAC: Adding query filter params (name=admin) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:01:07.935540 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4c17cc4-870d-4256-90a3-64f1457bd72d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:01:07.936897 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4c17cc4-870d-4256-90a3-64f1457bd72d None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:01:07.944682 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 582/1150] 10.222.0.77 () {56 vars in 1099 bytes} [Tue Aug 30 05:01:07 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:01:07.950325 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-413612f9-bd6c-4e2e-902d-0d4b55c36b47 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:01:07.959920 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:01:08.048505 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-413612f9-bd6c-4e2e-902d-0d4b55c36b47 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:01:08.080482 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7892a944-b761-4001-ad31-f063d33eece0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:01:08.081083 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7892a944-b761-4001-ad31-f063d33eece0 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:01:08.081359 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7892a944-b761-4001-ad31-f063d33eece0 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:01:08.081628 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7892a944-b761-4001-ad31-f063d33eece0 None admin] PATH_INFO: `/v3/users/2aa696d004424700ac807ab755d8a1c6` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:01:08.082710 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7892a944-b761-4001-ad31-f063d33eece0 None admin] RBAC: Authorizing `identity:get_user(user_id=2aa696d004424700ac807ab755d8a1c6)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:01:08.084651 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7892a944-b761-4001-ad31-f063d33eece0 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:01:08.086179 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 569/1151] 10.222.0.77 () {56 vars in 1177 bytes} [Tue Aug 30 05:01:07 2022] GET /identity/v3/users/2aa696d004424700ac807ab755d8a1c6 => generated 286 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:01:08.091727 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4c17cc4-870d-4256-90a3-64f1457bd72d None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:01:08.126110 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:01:08.188055 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4c17cc4-870d-4256-90a3-64f1457bd72d None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:01:08.220032 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37b97584-aaa4-4841-8130-323ca63af692 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:01:08.220636 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b97584-aaa4-4841-8130-323ca63af692 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:01:08.220909 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b97584-aaa4-4841-8130-323ca63af692 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:01:08.221174 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b97584-aaa4-4841-8130-323ca63af692 None admin] PATH_INFO: `/v3/projects/322586836c984b49b28c8c731d860632` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:01:08.222252 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37b97584-aaa4-4841-8130-323ca63af692 None admin] RBAC: Authorizing `identity:get_project(project_id=322586836c984b49b28c8c731d860632)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:01:08.223676 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37b97584-aaa4-4841-8130-323ca63af692 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:01:08.225853 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 583/1152] 10.222.0.77 () {56 vars in 1186 bytes} [Tue Aug 30 05:01:08 2022] GET /identity/v3/projects/322586836c984b49b28c8c731d860632 => generated 311 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:01:08.231854 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7892a944-b761-4001-ad31-f063d33eece0 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:01:08.302977 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:01:08.327703 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7892a944-b761-4001-ad31-f063d33eece0 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:01:08.360172 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa1a2b73-5877-4390-9ef9-6a684b496842 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:01:08.360801 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa1a2b73-5877-4390-9ef9-6a684b496842 None admin] REQUEST_METHOD: `PUT` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:01:08.361072 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa1a2b73-5877-4390-9ef9-6a684b496842 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:01:08.361338 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa1a2b73-5877-4390-9ef9-6a684b496842 None admin] PATH_INFO: `/v3/projects/322586836c984b49b28c8c731d860632/users/2aa696d004424700ac807ab755d8a1c6/roles/9d9d5d2c270146488249b7ed943280db` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:01:08.363193 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa1a2b73-5877-4390-9ef9-6a684b496842 None admin] RBAC: Authorizing `identity:create_grant(project_id=322586836c984b49b28c8c731d860632, user_id=2aa696d004424700ac807ab755d8a1c6, role_id=9d9d5d2c270146488249b7ed943280db)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:01:08.364603 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa1a2b73-5877-4390-9ef9-6a684b496842 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:01:08.377310 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 570/1153] 10.222.0.77 () {58 vars in 1439 bytes} [Tue Aug 30 05:01:08 2022] PUT /identity/v3/projects/322586836c984b49b28c8c731d860632/users/2aa696d004424700ac807ab755d8a1c6/roles/9d9d5d2c270146488249b7ed943280db => generated 0 bytes in 146 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:01:10.229968 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8362c885-b574-4ed6-9307-1d4908605d69 None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:01:10.230524 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8362c885-b574-4ed6-9307-1d4908605d69 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:01:10.230524 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8362c885-b574-4ed6-9307-1d4908605d69 None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:01:10.231953 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 584/1154] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:01:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:01:10.237428 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc6f8f73-aca8-4053-b0b9-d8b025688ee7 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:01:10.237846 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc6f8f73-aca8-4053-b0b9-d8b025688ee7 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:01:10.237967 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc6f8f73-aca8-4053-b0b9-d8b025688ee7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:01:10.256712 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-bc6f8f73-aca8-4053-b0b9-d8b025688ee7 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:01:10.257887 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-bc6f8f73-aca8-4053-b0b9-d8b025688ee7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:01:10.321914 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 571/1155] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:01:10 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 05:01:10.601530 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8362c885-b574-4ed6-9307-1d4908605d69 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:01:10.702766 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8362c885-b574-4ed6-9307-1d4908605d69 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:01:10.705063 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:01:10.736361 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2de91542-4a6e-4ab2-9012-3a6e97970536 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:01:10.736968 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2de91542-4a6e-4ab2-9012-3a6e97970536 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:01:10.737240 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2de91542-4a6e-4ab2-9012-3a6e97970536 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:01:10.737518 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2de91542-4a6e-4ab2-9012-3a6e97970536 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:01:10.748320 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2de91542-4a6e-4ab2-9012-3a6e97970536 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 05:01:10.753019 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-2de91542-4a6e-4ab2-9012-3a6e97970536 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 05:01:10.753019 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:01:10.753019 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:01:10.753019 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:01:10.753019 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:01:10.753019 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:01:10.753019 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:01:10.753019 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:01:10.753019 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:01:10.753019 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:01:10.753019 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:01:10.753019 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:01:10.753019 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 05:01:10.753019 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 05:01:10.753019 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 05:01:10.753019 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 05:01:10.754162 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 05:01:10.754162 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 05:01:10.754162 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 05:01:10.754162 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 05:01:10.754162 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:01:10.754162 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:01:10.754162 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:01:10.754162 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:01:10.754162 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:01:10.754162 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:01:10.754162 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:01:10.754162 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 05:01:10.754162 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:01:10.754162 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:01:10.754162 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:01:10.754162 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:01:10.755083 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:01:10.755083 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:01:10.755083 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:01:10.755083 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:01:10.755083 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 05:01:10.755083 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 05:01:10.755083 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 05:01:10.755083 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 05:01:10.755083 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 05:01:10.755083 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 05:01:10.755083 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 05:01:10.755083 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:01:10.755083 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 585/1156] 10.222.0.77 () {56 vars in 1096 bytes} [Tue Aug 30 05:01:10 2022] GET /identity/v3/roles/admin => generated 83 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:01:10.760449 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc6f8f73-aca8-4053-b0b9-d8b025688ee7 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:01:10.819170 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:01:10.856761 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc6f8f73-aca8-4053-b0b9-d8b025688ee7 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:01:10.890410 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6045fcff-95ac-4336-a66d-973ccde3c937 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:01:10.891027 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6045fcff-95ac-4336-a66d-973ccde3c937 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:01:10.891307 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6045fcff-95ac-4336-a66d-973ccde3c937 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:01:10.891577 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6045fcff-95ac-4336-a66d-973ccde3c937 None admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:01:10.892229 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6045fcff-95ac-4336-a66d-973ccde3c937 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:01:10.892526 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6045fcff-95ac-4336-a66d-973ccde3c937 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:01:10.893883 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6045fcff-95ac-4336-a66d-973ccde3c937 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:01:10.902345 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 572/1157] 10.222.0.77 () {56 vars in 1099 bytes} [Tue Aug 30 05:01:10 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:01:10.908388 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2de91542-4a6e-4ab2-9012-3a6e97970536 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:01:11.012294 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2de91542-4a6e-4ab2-9012-3a6e97970536 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:01:11.045566 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84f33e5a-236f-47cc-9b40-bb08b4c04b7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:01:11.046176 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f33e5a-236f-47cc-9b40-bb08b4c04b7d None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:01:11.046457 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f33e5a-236f-47cc-9b40-bb08b4c04b7d None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:01:11.046729 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f33e5a-236f-47cc-9b40-bb08b4c04b7d None admin] PATH_INFO: `/v3/users/2aa696d004424700ac807ab755d8a1c6` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:01:11.047863 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84f33e5a-236f-47cc-9b40-bb08b4c04b7d None admin] RBAC: Authorizing `identity:get_user(user_id=2aa696d004424700ac807ab755d8a1c6)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:01:11.049301 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84f33e5a-236f-47cc-9b40-bb08b4c04b7d None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:01:11.050887 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 586/1158] 10.222.0.77 () {56 vars in 1177 bytes} [Tue Aug 30 05:01:10 2022] GET /identity/v3/users/2aa696d004424700ac807ab755d8a1c6 => generated 286 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:01:11.056704 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6045fcff-95ac-4336-a66d-973ccde3c937 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:01:11.154030 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6045fcff-95ac-4336-a66d-973ccde3c937 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:01:11.178710 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:01:11.187413 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-863248ab-97d7-4e88-962a-d298d0717441 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:01:11.188002 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-863248ab-97d7-4e88-962a-d298d0717441 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:01:11.188282 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-863248ab-97d7-4e88-962a-d298d0717441 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:01:11.188555 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-863248ab-97d7-4e88-962a-d298d0717441 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:01:11.253197 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-863248ab-97d7-4e88-962a-d298d0717441 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:01:11.254629 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-863248ab-97d7-4e88-962a-d298d0717441 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:01:11.350338 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 573/1159] 10.222.0.77 () {58 vars in 1282 bytes} [Tue Aug 30 05:01:11 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 294 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 30 05:01:11.356634 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84f33e5a-236f-47cc-9b40-bb08b4c04b7d None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:01:11.387592 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:01:11.454766 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84f33e5a-236f-47cc-9b40-bb08b4c04b7d None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:01:11.488192 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14e27a4b-1722-48de-a729-ba2e1dba0f68 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:01:11.488797 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14e27a4b-1722-48de-a729-ba2e1dba0f68 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:01:11.489070 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14e27a4b-1722-48de-a729-ba2e1dba0f68 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:01:11.489335 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14e27a4b-1722-48de-a729-ba2e1dba0f68 None admin] PATH_INFO: `/v3/projects/322586836c984b49b28c8c731d860632` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:01:11.490406 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14e27a4b-1722-48de-a729-ba2e1dba0f68 None admin] RBAC: Authorizing `identity:get_project(project_id=322586836c984b49b28c8c731d860632)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:01:11.491841 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14e27a4b-1722-48de-a729-ba2e1dba0f68 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:01:11.493425 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 587/1160] 10.222.0.77 () {56 vars in 1186 bytes} [Tue Aug 30 05:01:11 2022] GET /identity/v3/projects/322586836c984b49b28c8c731d860632 => generated 311 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:01:11.499588 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-863248ab-97d7-4e88-962a-d298d0717441 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:01:11.526173 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:01:11.595622 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-863248ab-97d7-4e88-962a-d298d0717441 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:01:11.627640 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8578857-20dd-4576-9d58-98c5dbf52b61 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:01:11.628221 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8578857-20dd-4576-9d58-98c5dbf52b61 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:01:11.628498 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8578857-20dd-4576-9d58-98c5dbf52b61 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:01:11.628771 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8578857-20dd-4576-9d58-98c5dbf52b61 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:01:11.629478 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8578857-20dd-4576-9d58-98c5dbf52b61 None admin] RBAC: Adding query filter params (role.id=9d9d5d2c270146488249b7ed943280db, scope.project.id=322586836c984b49b28c8c731d860632, user.id=2aa696d004424700ac807ab755d8a1c6) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:01:11.629781 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8578857-20dd-4576-9d58-98c5dbf52b61 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:01:11.631719 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8578857-20dd-4576-9d58-98c5dbf52b61 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:01:11.637245 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 574/1161] 10.222.0.77 () {56 vars in 1374 bytes} [Tue Aug 30 05:01:11 2022] GET /identity/v3/role_assignments?user.id=2aa696d004424700ac807ab755d8a1c6&scope.project.id=322586836c984b49b28c8c731d860632&role.id=9d9d5d2c270146488249b7ed943280db => generated 608 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:01:13.544554 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1371ba3e-d0ba-4a1f-97af-99a09b0b4be4 None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:01:13.545055 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1371ba3e-d0ba-4a1f-97af-99a09b0b4be4 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:01:13.545150 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1371ba3e-d0ba-4a1f-97af-99a09b0b4be4 None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:01:13.546411 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 588/1162] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:01:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:01:13.551593 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15748639-c55b-4f5a-a46a-3440ac4ab3be None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:01:13.551971 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15748639-c55b-4f5a-a46a-3440ac4ab3be None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:01:13.552135 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15748639-c55b-4f5a-a46a-3440ac4ab3be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:01:13.570536 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-15748639-c55b-4f5a-a46a-3440ac4ab3be None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:01:13.571755 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-15748639-c55b-4f5a-a46a-3440ac4ab3be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:01:13.635925 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 575/1163] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:01:13 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 05:01:13.898512 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1371ba3e-d0ba-4a1f-97af-99a09b0b4be4 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:01:13.994768 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:01:13.998991 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1371ba3e-d0ba-4a1f-97af-99a09b0b4be4 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:01:14.032453 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cca69b1e-1565-4b06-9848-b7f77cee22e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:01:14.033041 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca69b1e-1565-4b06-9848-b7f77cee22e6 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:01:14.033314 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca69b1e-1565-4b06-9848-b7f77cee22e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:01:14.033579 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca69b1e-1565-4b06-9848-b7f77cee22e6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:01:14.097554 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cca69b1e-1565-4b06-9848-b7f77cee22e6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:01:14.099007 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cca69b1e-1565-4b06-9848-b7f77cee22e6 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:01:14.189583 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:01:14.193218 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 589/1164] 10.222.0.77 () {58 vars in 1282 bytes} [Tue Aug 30 05:01:13 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 295 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 30 05:01:14.199132 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15748639-c55b-4f5a-a46a-3440ac4ab3be None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:01:14.220012 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:01:14.296027 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15748639-c55b-4f5a-a46a-3440ac4ab3be None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:01:14.328703 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf7997af-ca71-4175-8b00-4153922dd848 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:01:14.329314 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf7997af-ca71-4175-8b00-4153922dd848 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:01:14.329594 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf7997af-ca71-4175-8b00-4153922dd848 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:01:14.329867 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf7997af-ca71-4175-8b00-4153922dd848 None admin] PATH_INFO: `/v3/domains/swift_test` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:01:14.336861 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf7997af-ca71-4175-8b00-4153922dd848 None admin] RBAC: Authorizing `identity:get_domain(domain_id=swift_test)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:01:14.338292 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf7997af-ca71-4175-8b00-4153922dd848 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:01:14.349378 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-cf7997af-ca71-4175-8b00-4153922dd848 None admin] Could not find domain: swift_test.: keystone.exception.DomainNotFound: Could not find domain: swift_test. Aug 30 05:01:14.349378 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:01:14.349378 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 712, in get_domain Aug 30 05:01:14.349378 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application project = self.driver.get_project(domain_id) Aug 30 05:01:14.349378 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 05:01:14.349378 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 05:01:14.349378 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 05:01:14.349378 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 05:01:14.349378 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: swift_test. Aug 30 05:01:14.349378 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:01:14.349378 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 05:01:14.349378 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:01:14.349378 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:01:14.349378 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:01:14.349378 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:01:14.349378 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:01:14.349378 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:01:14.350575 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:01:14.350575 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:01:14.350575 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:01:14.350575 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:01:14.350575 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:01:14.350575 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:01:14.350575 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 05:01:14.350575 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 05:01:14.350575 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 05:01:14.350575 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 05:01:14.350575 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:01:14.350575 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:01:14.350575 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:01:14.350575 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:01:14.350575 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:01:14.350575 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:01:14.351660 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:01:14.351660 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 05:01:14.351660 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:01:14.351660 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:01:14.351660 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:01:14.351660 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:01:14.351660 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:01:14.351660 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:01:14.351660 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:01:14.351660 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:01:14.351660 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 718, in get_domain Aug 30 05:01:14.351660 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 05:01:14.351660 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: swift_test. Aug 30 05:01:14.351660 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:01:14.351660 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 576/1165] 10.222.0.77 () {56 vars in 1117 bytes} [Tue Aug 30 05:01:14 2022] GET /identity/v3/domains/swift_test => generated 90 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:01:14.356714 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cca69b1e-1565-4b06-9848-b7f77cee22e6 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:01:14.452505 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cca69b1e-1565-4b06-9848-b7f77cee22e6 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:01:14.485445 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1945eeea-b47c-4a77-9e34-73bb2515f8b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:01:14.486031 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1945eeea-b47c-4a77-9e34-73bb2515f8b6 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:01:14.486303 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1945eeea-b47c-4a77-9e34-73bb2515f8b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:01:14.486566 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1945eeea-b47c-4a77-9e34-73bb2515f8b6 None admin] PATH_INFO: `/v3/domains` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:01:14.487216 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1945eeea-b47c-4a77-9e34-73bb2515f8b6 None admin] RBAC: Adding query filter params (name=swift_test) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:01:14.487509 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1945eeea-b47c-4a77-9e34-73bb2515f8b6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:01:14.488843 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1945eeea-b47c-4a77-9e34-73bb2515f8b6 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:01:14.495686 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 590/1166] 10.222.0.77 () {56 vars in 1115 bytes} [Tue Aug 30 05:01:14 2022] GET /identity/v3/domains?name=swift_test => generated 125 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:01:16.355319 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4150a8fc-029d-4398-9b03-e130b2ba9e3f None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:01:16.355890 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4150a8fc-029d-4398-9b03-e130b2ba9e3f None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:01:16.355890 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4150a8fc-029d-4398-9b03-e130b2ba9e3f None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:01:16.357154 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 577/1167] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:01:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:01:16.362154 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43eb6d18-6e0b-4e30-b80f-2a20c525db63 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:01:16.362582 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43eb6d18-6e0b-4e30-b80f-2a20c525db63 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:01:16.362686 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43eb6d18-6e0b-4e30-b80f-2a20c525db63 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:01:16.380727 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-43eb6d18-6e0b-4e30-b80f-2a20c525db63 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:01:16.382499 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-43eb6d18-6e0b-4e30-b80f-2a20c525db63 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:01:16.438552 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:01:16.447621 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 591/1168] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:01:16 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 05:01:16.710890 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4150a8fc-029d-4398-9b03-e130b2ba9e3f None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:01:16.789412 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:01:16.809129 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4150a8fc-029d-4398-9b03-e130b2ba9e3f None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:01:16.840458 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b01fd02-219b-44be-8a51-3778d0519377 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:01:16.841038 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b01fd02-219b-44be-8a51-3778d0519377 None admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:01:16.841312 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b01fd02-219b-44be-8a51-3778d0519377 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:01:16.841577 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b01fd02-219b-44be-8a51-3778d0519377 None admin] PATH_INFO: `/v3/domains` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:01:16.842231 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b01fd02-219b-44be-8a51-3778d0519377 None admin] RBAC: Authorizing `identity:create_domain()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:01:16.843610 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b01fd02-219b-44be-8a51-3778d0519377 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:01:16.859303 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 578/1169] 10.222.0.77 () {60 vars in 1138 bytes} [Tue Aug 30 05:01:16 2022] POST /identity/v3/domains => generated 267 bytes in 149 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:01:18.753176 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-798fbd4e-40e7-476c-9fbe-637d6eadfb8e None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:01:18.753738 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-798fbd4e-40e7-476c-9fbe-637d6eadfb8e None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:01:18.753738 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-798fbd4e-40e7-476c-9fbe-637d6eadfb8e None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:01:18.755006 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 592/1170] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:01:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:01:18.760881 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33106c45-f632-44a9-855e-8495e0fd82e2 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:01:18.761274 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33106c45-f632-44a9-855e-8495e0fd82e2 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:01:18.761436 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33106c45-f632-44a9-855e-8495e0fd82e2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:01:18.779427 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-33106c45-f632-44a9-855e-8495e0fd82e2 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:01:18.780610 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-33106c45-f632-44a9-855e-8495e0fd82e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:01:18.845179 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 579/1171] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:01:18 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 05:01:19.112556 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-798fbd4e-40e7-476c-9fbe-637d6eadfb8e None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:01:19.211148 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-798fbd4e-40e7-476c-9fbe-637d6eadfb8e None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:01:19.244237 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-995ab3b9-3aa8-48ec-8ed3-79ad808ccaca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:01:19.244845 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-995ab3b9-3aa8-48ec-8ed3-79ad808ccaca None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:01:19.245130 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-995ab3b9-3aa8-48ec-8ed3-79ad808ccaca None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:01:19.245402 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-995ab3b9-3aa8-48ec-8ed3-79ad808ccaca None admin] PATH_INFO: `/v3/domains/3ca1ae7b2c7c458faf79487dfeaf7661` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:01:19.255411 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-995ab3b9-3aa8-48ec-8ed3-79ad808ccaca None admin] RBAC: Authorizing `identity:get_domain(domain_id=3ca1ae7b2c7c458faf79487dfeaf7661)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:01:19.256852 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-995ab3b9-3aa8-48ec-8ed3-79ad808ccaca None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:01:19.258388 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 593/1172] 10.222.0.77 () {56 vars in 1183 bytes} [Tue Aug 30 05:01:19 2022] GET /identity/v3/domains/3ca1ae7b2c7c458faf79487dfeaf7661 => generated 267 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:01:19.263970 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33106c45-f632-44a9-855e-8495e0fd82e2 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:01:19.331620 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:01:19.363444 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33106c45-f632-44a9-855e-8495e0fd82e2 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:01:19.395315 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9680aa00-2ea6-412c-a2b2-f76e89ed32d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:01:19.395893 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9680aa00-2ea6-412c-a2b2-f76e89ed32d0 None admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:01:19.396164 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9680aa00-2ea6-412c-a2b2-f76e89ed32d0 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:01:19.396917 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9680aa00-2ea6-412c-a2b2-f76e89ed32d0 None admin] PATH_INFO: `/v3/projects` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:01:19.397553 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9680aa00-2ea6-412c-a2b2-f76e89ed32d0 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:01:19.398986 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9680aa00-2ea6-412c-a2b2-f76e89ed32d0 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:01:19.424061 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 580/1173] 10.222.0.77 () {60 vars in 1141 bytes} [Tue Aug 30 05:01:19 2022] POST /identity/v3/projects => generated 361 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:01:21.325692 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08be245a-2641-4307-b17c-e10a7e64b7dd None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:01:21.326243 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08be245a-2641-4307-b17c-e10a7e64b7dd None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:01:21.326243 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08be245a-2641-4307-b17c-e10a7e64b7dd None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:01:21.327576 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 594/1174] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:01:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:01:21.332824 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b9600d0-3448-46e0-9b93-044551c0f145 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:01:21.333260 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b9600d0-3448-46e0-9b93-044551c0f145 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:01:21.333363 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b9600d0-3448-46e0-9b93-044551c0f145 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:01:21.351743 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-8b9600d0-3448-46e0-9b93-044551c0f145 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:01:21.353205 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-8b9600d0-3448-46e0-9b93-044551c0f145 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:01:21.417601 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 581/1175] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:01:21 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 05:01:21.681473 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08be245a-2641-4307-b17c-e10a7e64b7dd None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:01:21.736466 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:01:21.781526 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08be245a-2641-4307-b17c-e10a7e64b7dd None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:01:21.813908 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44e0c2f4-af58-4e68-9c16-c7b40d582b94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:01:21.814509 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e0c2f4-af58-4e68-9c16-c7b40d582b94 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:01:21.814783 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e0c2f4-af58-4e68-9c16-c7b40d582b94 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:01:21.815085 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e0c2f4-af58-4e68-9c16-c7b40d582b94 None admin] PATH_INFO: `/v3/domains/3ca1ae7b2c7c458faf79487dfeaf7661` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:01:21.816167 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44e0c2f4-af58-4e68-9c16-c7b40d582b94 None admin] RBAC: Authorizing `identity:get_domain(domain_id=3ca1ae7b2c7c458faf79487dfeaf7661)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:01:21.817580 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44e0c2f4-af58-4e68-9c16-c7b40d582b94 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:01:21.819200 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 595/1176] 10.222.0.77 () {56 vars in 1183 bytes} [Tue Aug 30 05:01:21 2022] GET /identity/v3/domains/3ca1ae7b2c7c458faf79487dfeaf7661 => generated 267 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:01:21.824866 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b9600d0-3448-46e0-9b93-044551c0f145 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:01:21.874809 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:01:21.920287 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b9600d0-3448-46e0-9b93-044551c0f145 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:01:21.952508 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a62201c2-9543-4f9e-baf6-74ffb8bdfdfd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:01:21.953037 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a62201c2-9543-4f9e-baf6-74ffb8bdfdfd None admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:01:21.953311 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a62201c2-9543-4f9e-baf6-74ffb8bdfdfd None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:01:21.953579 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a62201c2-9543-4f9e-baf6-74ffb8bdfdfd None admin] PATH_INFO: `/v3/users` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:01:21.954204 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a62201c2-9543-4f9e-baf6-74ffb8bdfdfd None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:01:21.955644 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a62201c2-9543-4f9e-baf6-74ffb8bdfdfd None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:01:21.975745 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 582/1177] 10.222.0.77 () {60 vars in 1132 bytes} [Tue Aug 30 05:01:21 2022] POST /identity/v3/users => generated 311 bytes in 151 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:01:23.883521 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8af899c7-ef55-4f8c-bc7e-f9db3d99a820 None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:01:23.884112 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8af899c7-ef55-4f8c-bc7e-f9db3d99a820 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:01:23.884112 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8af899c7-ef55-4f8c-bc7e-f9db3d99a820 None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:01:23.885381 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 596/1178] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:01:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:01:23.890458 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-409b2924-1337-432f-bae9-38e62f5cc505 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:01:23.890901 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-409b2924-1337-432f-bae9-38e62f5cc505 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:01:23.891030 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-409b2924-1337-432f-bae9-38e62f5cc505 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:01:23.909173 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-409b2924-1337-432f-bae9-38e62f5cc505 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:01:23.910367 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-409b2924-1337-432f-bae9-38e62f5cc505 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:01:23.975182 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 583/1179] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:01:23 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 05:01:24.249511 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8af899c7-ef55-4f8c-bc7e-f9db3d99a820 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:01:24.348896 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8af899c7-ef55-4f8c-bc7e-f9db3d99a820 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:01:24.370176 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:01:24.382226 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c7f33e1-660b-43fc-9950-778a74e972e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:01:24.382856 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c7f33e1-660b-43fc-9950-778a74e972e9 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:01:24.383139 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c7f33e1-660b-43fc-9950-778a74e972e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:01:24.383407 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c7f33e1-660b-43fc-9950-778a74e972e9 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:01:24.393556 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c7f33e1-660b-43fc-9950-778a74e972e9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 05:01:24.398866 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-8c7f33e1-660b-43fc-9950-778a74e972e9 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 05:01:24.398866 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:01:24.398866 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:01:24.398866 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:01:24.398866 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:01:24.398866 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:01:24.398866 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:01:24.398866 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:01:24.398866 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:01:24.398866 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:01:24.398866 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:01:24.398866 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:01:24.398866 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 05:01:24.398866 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 05:01:24.398866 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 05:01:24.398866 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 05:01:24.400016 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 05:01:24.400016 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 05:01:24.400016 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 05:01:24.400016 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 05:01:24.400016 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:01:24.400016 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:01:24.400016 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:01:24.400016 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:01:24.400016 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:01:24.400016 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:01:24.400016 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:01:24.400016 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 05:01:24.400016 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:01:24.400016 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:01:24.400016 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:01:24.400016 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:01:24.401150 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:01:24.401150 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:01:24.401150 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:01:24.401150 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:01:24.401150 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 05:01:24.401150 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 05:01:24.401150 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 05:01:24.401150 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 05:01:24.401150 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 05:01:24.401150 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 05:01:24.401150 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 05:01:24.401150 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:01:24.401150 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 597/1180] 10.222.0.77 () {56 vars in 1096 bytes} [Tue Aug 30 05:01:24 2022] GET /identity/v3/roles/admin => generated 83 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:01:24.405770 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-409b2924-1337-432f-bae9-38e62f5cc505 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:01:24.444812 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:01:24.504266 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-409b2924-1337-432f-bae9-38e62f5cc505 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:01:24.536954 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6bf9db9-5c0a-42ec-9146-41b0e8cf53b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:01:24.537545 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6bf9db9-5c0a-42ec-9146-41b0e8cf53b2 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:01:24.537826 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6bf9db9-5c0a-42ec-9146-41b0e8cf53b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:01:24.538099 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6bf9db9-5c0a-42ec-9146-41b0e8cf53b2 None admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:01:24.538752 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6bf9db9-5c0a-42ec-9146-41b0e8cf53b2 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:01:24.539105 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6bf9db9-5c0a-42ec-9146-41b0e8cf53b2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:01:24.540464 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6bf9db9-5c0a-42ec-9146-41b0e8cf53b2 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:01:24.548252 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 584/1181] 10.222.0.77 () {56 vars in 1099 bytes} [Tue Aug 30 05:01:24 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:01:24.554640 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c7f33e1-660b-43fc-9950-778a74e972e9 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:01:24.652038 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c7f33e1-660b-43fc-9950-778a74e972e9 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:01:24.684355 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30a42c87-fb96-4880-9bb6-4d6844b9c24c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:01:24.684956 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30a42c87-fb96-4880-9bb6-4d6844b9c24c None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:01:24.685234 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30a42c87-fb96-4880-9bb6-4d6844b9c24c None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:01:24.685502 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30a42c87-fb96-4880-9bb6-4d6844b9c24c None admin] PATH_INFO: `/v3/users/48edfb54064e473c86ef2f5be11664b3` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:01:24.698616 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30a42c87-fb96-4880-9bb6-4d6844b9c24c None admin] RBAC: Authorizing `identity:get_user(user_id=48edfb54064e473c86ef2f5be11664b3)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:01:24.700085 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30a42c87-fb96-4880-9bb6-4d6844b9c24c None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:01:24.701634 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 598/1182] 10.222.0.77 () {56 vars in 1177 bytes} [Tue Aug 30 05:01:24 2022] GET /identity/v3/users/48edfb54064e473c86ef2f5be11664b3 => generated 311 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:01:24.707244 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6bf9db9-5c0a-42ec-9146-41b0e8cf53b2 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:01:24.802336 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:01:24.805979 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6bf9db9-5c0a-42ec-9146-41b0e8cf53b2 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:01:24.837944 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ace6b16-0892-4db0-b3ac-cea3987a2eb0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:01:24.838593 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ace6b16-0892-4db0-b3ac-cea3987a2eb0 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:01:24.838894 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ace6b16-0892-4db0-b3ac-cea3987a2eb0 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:01:24.839166 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ace6b16-0892-4db0-b3ac-cea3987a2eb0 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:01:24.904087 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ace6b16-0892-4db0-b3ac-cea3987a2eb0 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:01:24.905712 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ace6b16-0892-4db0-b3ac-cea3987a2eb0 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:01:24.999273 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:01:25.003042 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 585/1183] 10.222.0.77 () {58 vars in 1282 bytes} [Tue Aug 30 05:01:24 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 296 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 30 05:01:25.008691 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30a42c87-fb96-4880-9bb6-4d6844b9c24c None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:01:25.049278 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:01:25.106369 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30a42c87-fb96-4880-9bb6-4d6844b9c24c None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:01:25.138909 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f7674e9-6c74-43cc-be04-53efb4c5ff29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:01:25.139515 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f7674e9-6c74-43cc-be04-53efb4c5ff29 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:01:25.139789 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f7674e9-6c74-43cc-be04-53efb4c5ff29 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:01:25.140055 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f7674e9-6c74-43cc-be04-53efb4c5ff29 None admin] PATH_INFO: `/v3/projects/2ca6972c3c194291bb03a1bd39a05185` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:01:25.141132 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f7674e9-6c74-43cc-be04-53efb4c5ff29 None admin] RBAC: Authorizing `identity:get_project(project_id=2ca6972c3c194291bb03a1bd39a05185)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:01:25.142547 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f7674e9-6c74-43cc-be04-53efb4c5ff29 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:01:25.144159 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 599/1184] 10.222.0.77 () {56 vars in 1186 bytes} [Tue Aug 30 05:01:25 2022] GET /identity/v3/projects/2ca6972c3c194291bb03a1bd39a05185 => generated 361 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:01:25.150731 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ace6b16-0892-4db0-b3ac-cea3987a2eb0 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:01:25.245319 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ace6b16-0892-4db0-b3ac-cea3987a2eb0 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:01:25.277881 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dd7ac27-b955-4be4-85fc-10cfb32f4626 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:01:25.278459 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dd7ac27-b955-4be4-85fc-10cfb32f4626 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:01:25.278734 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dd7ac27-b955-4be4-85fc-10cfb32f4626 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:01:25.279028 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dd7ac27-b955-4be4-85fc-10cfb32f4626 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:01:25.279732 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dd7ac27-b955-4be4-85fc-10cfb32f4626 None admin] RBAC: Adding query filter params (role.id=9d9d5d2c270146488249b7ed943280db, scope.project.id=2ca6972c3c194291bb03a1bd39a05185, user.id=48edfb54064e473c86ef2f5be11664b3) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:01:25.280032 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dd7ac27-b955-4be4-85fc-10cfb32f4626 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:01:25.281466 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dd7ac27-b955-4be4-85fc-10cfb32f4626 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:01:25.286735 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 586/1185] 10.222.0.77 () {56 vars in 1374 bytes} [Tue Aug 30 05:01:25 2022] GET /identity/v3/role_assignments?user.id=48edfb54064e473c86ef2f5be11664b3&scope.project.id=2ca6972c3c194291bb03a1bd39a05185&role.id=9d9d5d2c270146488249b7ed943280db => generated 259 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:01:27.130645 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b70e075-3c79-4d33-8ef6-879c7be977df None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:01:27.131192 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b70e075-3c79-4d33-8ef6-879c7be977df None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:01:27.131192 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b70e075-3c79-4d33-8ef6-879c7be977df None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:01:27.132528 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 600/1186] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:01:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:01:27.137907 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecd8df3a-88c0-461f-ba9e-427b68862731 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:01:27.138319 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecd8df3a-88c0-461f-ba9e-427b68862731 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:01:27.138454 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecd8df3a-88c0-461f-ba9e-427b68862731 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:01:27.156092 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-ecd8df3a-88c0-461f-ba9e-427b68862731 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:01:27.157273 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-ecd8df3a-88c0-461f-ba9e-427b68862731 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:01:27.214671 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:01:27.223322 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 587/1187] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:01:27 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 05:01:27.488048 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b70e075-3c79-4d33-8ef6-879c7be977df None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:01:27.586042 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b70e075-3c79-4d33-8ef6-879c7be977df None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:01:27.594798 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:01:27.619729 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c2703aa-17d6-42e0-82b9-1224032f36a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:01:27.620328 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c2703aa-17d6-42e0-82b9-1224032f36a9 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:01:27.620602 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c2703aa-17d6-42e0-82b9-1224032f36a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:01:27.620866 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c2703aa-17d6-42e0-82b9-1224032f36a9 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:01:27.631228 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c2703aa-17d6-42e0-82b9-1224032f36a9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 05:01:27.635952 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-2c2703aa-17d6-42e0-82b9-1224032f36a9 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 05:01:27.635952 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:01:27.635952 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:01:27.635952 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:01:27.635952 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:01:27.635952 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:01:27.635952 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:01:27.635952 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:01:27.635952 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:01:27.635952 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:01:27.635952 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:01:27.635952 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:01:27.635952 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 05:01:27.635952 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 05:01:27.635952 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 05:01:27.635952 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 05:01:27.637044 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 05:01:27.637044 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 05:01:27.637044 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 05:01:27.637044 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 05:01:27.637044 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:01:27.637044 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:01:27.637044 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:01:27.637044 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:01:27.637044 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:01:27.637044 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:01:27.637044 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:01:27.637044 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 05:01:27.637044 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:01:27.637044 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:01:27.637044 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:01:27.637044 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:01:27.637954 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:01:27.637954 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:01:27.637954 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:01:27.637954 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:01:27.637954 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 05:01:27.637954 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 05:01:27.637954 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 05:01:27.637954 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 05:01:27.637954 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 05:01:27.637954 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 05:01:27.637954 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 05:01:27.637954 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:01:27.637954 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 601/1188] 10.222.0.77 () {56 vars in 1096 bytes} [Tue Aug 30 05:01:27 2022] GET /identity/v3/roles/admin => generated 83 bytes in 150 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:01:27.642805 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecd8df3a-88c0-461f-ba9e-427b68862731 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:01:27.737807 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecd8df3a-88c0-461f-ba9e-427b68862731 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:01:27.770930 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-275012f2-942f-4f86-afc2-f64ebc322ea2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:01:27.771541 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-275012f2-942f-4f86-afc2-f64ebc322ea2 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:01:27.771818 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-275012f2-942f-4f86-afc2-f64ebc322ea2 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:01:27.772085 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-275012f2-942f-4f86-afc2-f64ebc322ea2 None admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:01:27.772729 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-275012f2-942f-4f86-afc2-f64ebc322ea2 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:01:27.773025 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-275012f2-942f-4f86-afc2-f64ebc322ea2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:01:27.774372 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-275012f2-942f-4f86-afc2-f64ebc322ea2 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:01:27.782085 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 588/1189] 10.222.0.77 () {56 vars in 1099 bytes} [Tue Aug 30 05:01:27 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:01:27.788067 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c2703aa-17d6-42e0-82b9-1224032f36a9 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:01:27.884407 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c2703aa-17d6-42e0-82b9-1224032f36a9 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:01:27.916685 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abdbf0d7-8673-40af-bb36-645b538e1d99 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:01:27.917310 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abdbf0d7-8673-40af-bb36-645b538e1d99 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:01:27.917579 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abdbf0d7-8673-40af-bb36-645b538e1d99 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:01:27.917841 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abdbf0d7-8673-40af-bb36-645b538e1d99 None admin] PATH_INFO: `/v3/users/48edfb54064e473c86ef2f5be11664b3` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:01:27.918947 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abdbf0d7-8673-40af-bb36-645b538e1d99 None admin] RBAC: Authorizing `identity:get_user(user_id=48edfb54064e473c86ef2f5be11664b3)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:01:27.920351 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abdbf0d7-8673-40af-bb36-645b538e1d99 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:01:27.921882 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 602/1190] 10.222.0.77 () {56 vars in 1177 bytes} [Tue Aug 30 05:01:27 2022] GET /identity/v3/users/48edfb54064e473c86ef2f5be11664b3 => generated 311 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:01:27.928002 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-275012f2-942f-4f86-afc2-f64ebc322ea2 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:01:27.982499 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:01:28.025494 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-275012f2-942f-4f86-afc2-f64ebc322ea2 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:01:28.058277 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9c09ad6-74e5-4032-ba68-66df9fa4290f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:01:28.058918 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9c09ad6-74e5-4032-ba68-66df9fa4290f None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:01:28.059201 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9c09ad6-74e5-4032-ba68-66df9fa4290f None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:01:28.059473 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9c09ad6-74e5-4032-ba68-66df9fa4290f None admin] PATH_INFO: `/v3/projects/2ca6972c3c194291bb03a1bd39a05185` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:01:28.060571 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9c09ad6-74e5-4032-ba68-66df9fa4290f None admin] RBAC: Authorizing `identity:get_project(project_id=2ca6972c3c194291bb03a1bd39a05185)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:01:28.061995 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9c09ad6-74e5-4032-ba68-66df9fa4290f None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:01:28.063619 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 589/1191] 10.222.0.77 () {56 vars in 1186 bytes} [Tue Aug 30 05:01:27 2022] GET /identity/v3/projects/2ca6972c3c194291bb03a1bd39a05185 => generated 361 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:01:28.069248 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abdbf0d7-8673-40af-bb36-645b538e1d99 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:01:28.102965 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:01:28.165791 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abdbf0d7-8673-40af-bb36-645b538e1d99 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:01:28.198750 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57a794b1-4e24-4249-bcd6-bd77aacc6840 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:01:28.199421 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57a794b1-4e24-4249-bcd6-bd77aacc6840 None admin] REQUEST_METHOD: `PUT` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:01:28.199698 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57a794b1-4e24-4249-bcd6-bd77aacc6840 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:01:28.199969 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57a794b1-4e24-4249-bcd6-bd77aacc6840 None admin] PATH_INFO: `/v3/projects/2ca6972c3c194291bb03a1bd39a05185/users/48edfb54064e473c86ef2f5be11664b3/roles/9d9d5d2c270146488249b7ed943280db` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:01:28.201831 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57a794b1-4e24-4249-bcd6-bd77aacc6840 None admin] RBAC: Authorizing `identity:create_grant(project_id=2ca6972c3c194291bb03a1bd39a05185, user_id=48edfb54064e473c86ef2f5be11664b3, role_id=9d9d5d2c270146488249b7ed943280db)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:01:28.203288 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57a794b1-4e24-4249-bcd6-bd77aacc6840 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:01:28.215839 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 603/1192] 10.222.0.77 () {58 vars in 1439 bytes} [Tue Aug 30 05:01:28 2022] PUT /identity/v3/projects/2ca6972c3c194291bb03a1bd39a05185/users/48edfb54064e473c86ef2f5be11664b3/roles/9d9d5d2c270146488249b7ed943280db => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:01:30.084234 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c6cf932-d063-44ef-bded-a117dc28ea79 None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:01:30.084780 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c6cf932-d063-44ef-bded-a117dc28ea79 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:01:30.084780 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c6cf932-d063-44ef-bded-a117dc28ea79 None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:01:30.086128 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 590/1193] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:01:30 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:01:30.091274 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30dd0df0-707d-4d0e-8e24-034dae896a48 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:01:30.091709 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30dd0df0-707d-4d0e-8e24-034dae896a48 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:01:30.091811 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30dd0df0-707d-4d0e-8e24-034dae896a48 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:01:30.109674 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-30dd0df0-707d-4d0e-8e24-034dae896a48 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:01:30.110872 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-30dd0df0-707d-4d0e-8e24-034dae896a48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:01:30.175531 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 604/1194] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:01:30 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 05:01:30.447934 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c6cf932-d063-44ef-bded-a117dc28ea79 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:01:30.547110 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c6cf932-d063-44ef-bded-a117dc28ea79 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:01:30.570082 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:01:30.580852 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a225b3a-53a9-4459-8678-6a8f45c0f8f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:01:30.581464 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a225b3a-53a9-4459-8678-6a8f45c0f8f4 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:01:30.581742 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a225b3a-53a9-4459-8678-6a8f45c0f8f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:01:30.582014 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a225b3a-53a9-4459-8678-6a8f45c0f8f4 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:01:30.592274 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a225b3a-53a9-4459-8678-6a8f45c0f8f4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 05:01:30.597033 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-1a225b3a-53a9-4459-8678-6a8f45c0f8f4 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 05:01:30.597033 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:01:30.597033 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:01:30.597033 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:01:30.597033 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:01:30.597033 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:01:30.597033 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:01:30.597033 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:01:30.597033 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:01:30.597033 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:01:30.597033 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:01:30.597033 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:01:30.597033 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 05:01:30.597033 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 05:01:30.597033 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 05:01:30.597033 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 05:01:30.598104 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 05:01:30.598104 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 05:01:30.598104 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 05:01:30.598104 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 05:01:30.598104 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:01:30.598104 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:01:30.598104 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:01:30.598104 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:01:30.598104 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:01:30.598104 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:01:30.598104 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:01:30.598104 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 05:01:30.598104 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:01:30.598104 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:01:30.598104 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:01:30.598104 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:01:30.599190 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:01:30.599190 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:01:30.599190 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:01:30.599190 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:01:30.599190 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 05:01:30.599190 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 05:01:30.599190 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 05:01:30.599190 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 05:01:30.599190 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 05:01:30.599190 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 05:01:30.599190 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 05:01:30.599190 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:01:30.599190 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 591/1195] 10.222.0.77 () {56 vars in 1096 bytes} [Tue Aug 30 05:01:30 2022] GET /identity/v3/roles/admin => generated 83 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:01:30.604493 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30dd0df0-707d-4d0e-8e24-034dae896a48 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:01:30.631171 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:01:30.701838 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30dd0df0-707d-4d0e-8e24-034dae896a48 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:01:30.734098 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83c45113-482a-4182-bb45-ebb62107ca80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:01:30.734684 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83c45113-482a-4182-bb45-ebb62107ca80 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:01:30.734990 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83c45113-482a-4182-bb45-ebb62107ca80 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:01:30.735260 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83c45113-482a-4182-bb45-ebb62107ca80 None admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:01:30.735914 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83c45113-482a-4182-bb45-ebb62107ca80 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:01:30.736211 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83c45113-482a-4182-bb45-ebb62107ca80 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:01:30.737579 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83c45113-482a-4182-bb45-ebb62107ca80 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:01:30.746207 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 605/1196] 10.222.0.77 () {56 vars in 1099 bytes} [Tue Aug 30 05:01:30 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:01:30.752216 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a225b3a-53a9-4459-8678-6a8f45c0f8f4 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:01:30.853252 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a225b3a-53a9-4459-8678-6a8f45c0f8f4 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:01:30.885742 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a55af440-f34e-4408-8533-dbedacb23942 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:01:30.886352 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a55af440-f34e-4408-8533-dbedacb23942 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:01:30.886623 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a55af440-f34e-4408-8533-dbedacb23942 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:01:30.886915 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a55af440-f34e-4408-8533-dbedacb23942 None admin] PATH_INFO: `/v3/users/48edfb54064e473c86ef2f5be11664b3` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:01:30.887999 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a55af440-f34e-4408-8533-dbedacb23942 None admin] RBAC: Authorizing `identity:get_user(user_id=48edfb54064e473c86ef2f5be11664b3)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:01:30.889420 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a55af440-f34e-4408-8533-dbedacb23942 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:01:30.890997 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 592/1197] 10.222.0.77 () {56 vars in 1177 bytes} [Tue Aug 30 05:01:30 2022] GET /identity/v3/users/48edfb54064e473c86ef2f5be11664b3 => generated 311 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:01:30.897074 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83c45113-482a-4182-bb45-ebb62107ca80 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:01:30.982878 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:01:30.995757 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83c45113-482a-4182-bb45-ebb62107ca80 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:01:31.027740 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96b8340f-541e-4534-a394-21365124dc5e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:01:31.028327 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b8340f-541e-4534-a394-21365124dc5e None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:01:31.028601 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b8340f-541e-4534-a394-21365124dc5e None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:01:31.028867 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b8340f-541e-4534-a394-21365124dc5e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:01:31.093088 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96b8340f-541e-4534-a394-21365124dc5e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:01:31.094521 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96b8340f-541e-4534-a394-21365124dc5e None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:01:31.178177 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:01:31.190793 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 606/1198] 10.222.0.77 () {58 vars in 1282 bytes} [Tue Aug 30 05:01:30 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 294 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 30 05:01:31.196717 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a55af440-f34e-4408-8533-dbedacb23942 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:01:31.249092 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:01:31.295389 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a55af440-f34e-4408-8533-dbedacb23942 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:01:31.327093 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a183146e-6925-4c9b-9631-0717628726ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:01:31.327699 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a183146e-6925-4c9b-9631-0717628726ba None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:01:31.327979 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a183146e-6925-4c9b-9631-0717628726ba None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:01:31.328250 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a183146e-6925-4c9b-9631-0717628726ba None admin] PATH_INFO: `/v3/projects/2ca6972c3c194291bb03a1bd39a05185` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:01:31.329349 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a183146e-6925-4c9b-9631-0717628726ba None admin] RBAC: Authorizing `identity:get_project(project_id=2ca6972c3c194291bb03a1bd39a05185)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:01:31.330773 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a183146e-6925-4c9b-9631-0717628726ba None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:01:31.332407 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 593/1199] 10.222.0.77 () {56 vars in 1186 bytes} [Tue Aug 30 05:01:31 2022] GET /identity/v3/projects/2ca6972c3c194291bb03a1bd39a05185 => generated 361 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:01:31.338869 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96b8340f-541e-4534-a394-21365124dc5e None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:01:31.435515 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96b8340f-541e-4534-a394-21365124dc5e None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:01:31.468950 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6a65280-d7b9-4948-89fe-cbc4e945d1e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:01:31.469544 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6a65280-d7b9-4948-89fe-cbc4e945d1e2 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:01:31.469823 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6a65280-d7b9-4948-89fe-cbc4e945d1e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:01:31.470093 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6a65280-d7b9-4948-89fe-cbc4e945d1e2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:01:31.470804 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6a65280-d7b9-4948-89fe-cbc4e945d1e2 None admin] RBAC: Adding query filter params (role.id=9d9d5d2c270146488249b7ed943280db, scope.project.id=2ca6972c3c194291bb03a1bd39a05185, user.id=48edfb54064e473c86ef2f5be11664b3) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:01:31.471134 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6a65280-d7b9-4948-89fe-cbc4e945d1e2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:01:31.472546 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6a65280-d7b9-4948-89fe-cbc4e945d1e2 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:01:31.479057 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 607/1200] 10.222.0.77 () {56 vars in 1374 bytes} [Tue Aug 30 05:01:31 2022] GET /identity/v3/role_assignments?user.id=48edfb54064e473c86ef2f5be11664b3&scope.project.id=2ca6972c3c194291bb03a1bd39a05185&role.id=9d9d5d2c270146488249b7ed943280db => generated 608 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:01:41.323015 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=40 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:01:41.476192 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=74 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:02:26.279899 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de0c37ff-5c05-487d-a000-cdb47a6fda83 None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:26.280494 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de0c37ff-5c05-487d-a000-cdb47a6fda83 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:26.280494 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de0c37ff-5c05-487d-a000-cdb47a6fda83 None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:26.282238 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 594/1201] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:02:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:02:26.288143 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed70f473-5518-413e-8513-a970a95134d0 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:26.288608 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed70f473-5518-413e-8513-a970a95134d0 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:26.288812 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed70f473-5518-413e-8513-a970a95134d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:26.315719 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-ed70f473-5518-413e-8513-a970a95134d0 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:02:26.316983 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-ed70f473-5518-413e-8513-a970a95134d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:02:26.398465 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 608/1202] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:02:26 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 05:02:26.699091 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de0c37ff-5c05-487d-a000-cdb47a6fda83 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:02:26.833069 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de0c37ff-5c05-487d-a000-cdb47a6fda83 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:02:26.875032 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-360d1267-0a3d-484e-952f-d0ee5561b3d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:02:26.875663 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-360d1267-0a3d-484e-952f-d0ee5561b3d8 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:26.875936 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-360d1267-0a3d-484e-952f-d0ee5561b3d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:26.876213 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-360d1267-0a3d-484e-952f-d0ee5561b3d8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:26.887945 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-360d1267-0a3d-484e-952f-d0ee5561b3d8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:02:26.889403 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-360d1267-0a3d-484e-952f-d0ee5561b3d8 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:02:26.905031 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-360d1267-0a3d-484e-952f-d0ee5561b3d8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 05:02:26.905031 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:02:26.905031 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:02:26.905031 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:02:26.905031 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:02:26.905031 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:02:26.905031 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:02:26.905031 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:02:26.905031 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:02:26.905031 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:02:26.905031 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:02:26.905031 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:02:26.905031 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 05:02:26.905031 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 05:02:26.905031 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 05:02:26.905031 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 05:02:26.906415 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:02:26.906415 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:02:26.906415 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:02:26.906415 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:02:26.906415 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:02:26.906415 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:02:26.906415 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:02:26.906415 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 05:02:26.906415 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:02:26.906415 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:02:26.906415 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:02:26.906415 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:02:26.906415 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:02:26.906415 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:02:26.906415 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:02:26.906415 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:02:26.906415 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 05:02:26.907644 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 05:02:26.907644 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 05:02:26.907644 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:02:26.907644 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 595/1203] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 05:02:26 2022] GET /identity/v3/domains/Default => generated 87 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:02:26.913104 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed70f473-5518-413e-8513-a970a95134d0 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:02:27.046045 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed70f473-5518-413e-8513-a970a95134d0 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:02:27.083127 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:02:27.090205 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19f93b92-c9af-47b1-94a8-d0f31120889d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:02:27.090894 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19f93b92-c9af-47b1-94a8-d0f31120889d None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:27.092127 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19f93b92-c9af-47b1-94a8-d0f31120889d None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:27.092127 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19f93b92-c9af-47b1-94a8-d0f31120889d None admin] PATH_INFO: `/v3/domains` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:27.092127 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19f93b92-c9af-47b1-94a8-d0f31120889d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:02:27.092363 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19f93b92-c9af-47b1-94a8-d0f31120889d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:02:27.093746 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19f93b92-c9af-47b1-94a8-d0f31120889d None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:02:27.112677 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 609/1204] 10.222.0.77 () {56 vars in 1109 bytes} [Tue Aug 30 05:02:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:02:27.114849 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-360d1267-0a3d-484e-952f-d0ee5561b3d8 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:02:27.184743 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:02:27.246909 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-360d1267-0a3d-484e-952f-d0ee5561b3d8 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:02:27.297568 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d694376f-3ff4-44f0-8394-fc3a93deb240 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:02:27.298191 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d694376f-3ff4-44f0-8394-fc3a93deb240 None admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:27.298423 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d694376f-3ff4-44f0-8394-fc3a93deb240 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:27.298758 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d694376f-3ff4-44f0-8394-fc3a93deb240 None admin] PATH_INFO: `/v3/users` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:27.299744 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d694376f-3ff4-44f0-8394-fc3a93deb240 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:02:27.301897 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d694376f-3ff4-44f0-8394-fc3a93deb240 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:02:27.330858 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 596/1205] 10.222.0.77 () {60 vars in 1132 bytes} [Tue Aug 30 05:02:27 2022] POST /identity/v3/users => generated 251 bytes in 214 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:02:29.446880 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e19743a-e91f-4f17-9bf3-be97d9daf7ca None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:29.447427 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e19743a-e91f-4f17-9bf3-be97d9daf7ca None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:29.447427 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e19743a-e91f-4f17-9bf3-be97d9daf7ca None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:29.449049 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 610/1206] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:02:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:02:29.454846 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34190012-c5ff-497a-89df-03fff1415759 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:29.455339 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34190012-c5ff-497a-89df-03fff1415759 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:29.455428 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34190012-c5ff-497a-89df-03fff1415759 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:29.478942 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-34190012-c5ff-497a-89df-03fff1415759 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:02:29.480133 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-34190012-c5ff-497a-89df-03fff1415759 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:02:29.565330 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 597/1207] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:02:29 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 30 05:02:29.868798 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e19743a-e91f-4f17-9bf3-be97d9daf7ca None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:02:30.000011 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e19743a-e91f-4f17-9bf3-be97d9daf7ca None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:02:30.049907 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-511a93f5-8ccf-4723-b4b2-82600a89129f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:02:30.050866 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-511a93f5-8ccf-4723-b4b2-82600a89129f None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:30.051307 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-511a93f5-8ccf-4723-b4b2-82600a89129f None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:30.051730 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-511a93f5-8ccf-4723-b4b2-82600a89129f None admin] PATH_INFO: `/v3/roles/service` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:30.070038 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-511a93f5-8ccf-4723-b4b2-82600a89129f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 05:02:30.078877 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-511a93f5-8ccf-4723-b4b2-82600a89129f None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 30 05:02:30.078877 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:02:30.078877 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:02:30.078877 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:02:30.078877 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:02:30.078877 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:02:30.078877 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:02:30.078877 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:02:30.078877 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:02:30.078877 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:02:30.078877 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:02:30.078877 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:02:30.078877 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 05:02:30.078877 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 05:02:30.078877 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 05:02:30.078877 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 05:02:30.079943 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 05:02:30.079943 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 05:02:30.079943 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 05:02:30.079943 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 05:02:30.079943 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:02:30.079943 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:02:30.079943 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:02:30.079943 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:02:30.079943 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:02:30.079943 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:02:30.079943 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:02:30.079943 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 05:02:30.079943 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:02:30.079943 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:02:30.079943 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:02:30.079943 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:02:30.086998 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:02:30.086998 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:02:30.086998 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:02:30.086998 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:02:30.086998 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 05:02:30.086998 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 05:02:30.086998 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 05:02:30.086998 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 05:02:30.086998 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 05:02:30.086998 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 05:02:30.086998 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 30 05:02:30.086998 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:02:30.086998 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 611/1208] 10.222.0.77 () {56 vars in 1102 bytes} [Tue Aug 30 05:02:29 2022] GET /identity/v3/roles/service => generated 85 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:02:30.089616 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34190012-c5ff-497a-89df-03fff1415759 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:02:30.145025 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:02:30.219392 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34190012-c5ff-497a-89df-03fff1415759 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:02:30.268608 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a7014a-8720-4e93-9111-8a2fbe4a8a05 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:02:30.269217 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a7014a-8720-4e93-9111-8a2fbe4a8a05 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:30.269494 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a7014a-8720-4e93-9111-8a2fbe4a8a05 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:30.269849 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a7014a-8720-4e93-9111-8a2fbe4a8a05 None admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:30.270593 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80a7014a-8720-4e93-9111-8a2fbe4a8a05 None admin] RBAC: Adding query filter params (name=service) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:02:30.271045 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80a7014a-8720-4e93-9111-8a2fbe4a8a05 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:02:30.272294 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80a7014a-8720-4e93-9111-8a2fbe4a8a05 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:02:30.280941 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 598/1209] 10.222.0.77 () {56 vars in 1103 bytes} [Tue Aug 30 05:02:30 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:02:30.287613 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-511a93f5-8ccf-4723-b4b2-82600a89129f None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:02:30.359897 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:02:30.419771 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-511a93f5-8ccf-4723-b4b2-82600a89129f None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:02:30.466014 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f39fb155-1d57-4af1-b2f8-d01b2f1bd770 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:02:30.466489 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f39fb155-1d57-4af1-b2f8-d01b2f1bd770 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:30.466732 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f39fb155-1d57-4af1-b2f8-d01b2f1bd770 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:30.467010 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f39fb155-1d57-4af1-b2f8-d01b2f1bd770 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:30.481081 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f39fb155-1d57-4af1-b2f8-d01b2f1bd770 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:02:30.485295 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f39fb155-1d57-4af1-b2f8-d01b2f1bd770 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:02:30.502610 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-f39fb155-1d57-4af1-b2f8-d01b2f1bd770 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 05:02:30.502610 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:02:30.502610 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:02:30.502610 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:02:30.502610 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:02:30.502610 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:02:30.502610 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:02:30.502610 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:02:30.502610 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:02:30.502610 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:02:30.502610 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:02:30.502610 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:02:30.502610 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 05:02:30.502610 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 05:02:30.502610 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 05:02:30.502610 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 05:02:30.503936 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:02:30.503936 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:02:30.503936 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:02:30.503936 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:02:30.503936 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:02:30.503936 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:02:30.503936 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:02:30.503936 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 05:02:30.503936 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:02:30.503936 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:02:30.503936 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:02:30.503936 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:02:30.503936 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:02:30.503936 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:02:30.503936 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:02:30.503936 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:02:30.503936 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 05:02:30.505050 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 05:02:30.505050 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 05:02:30.505050 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:02:30.505050 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 612/1210] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 05:02:30 2022] GET /identity/v3/domains/Default => generated 87 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:02:30.512706 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a7014a-8720-4e93-9111-8a2fbe4a8a05 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:02:30.641837 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a7014a-8720-4e93-9111-8a2fbe4a8a05 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:02:30.643626 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:02:30.683196 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70b04179-1148-4d0f-b8a1-6df4f9d84c30 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:02:30.683196 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70b04179-1148-4d0f-b8a1-6df4f9d84c30 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:30.683488 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70b04179-1148-4d0f-b8a1-6df4f9d84c30 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:30.683570 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70b04179-1148-4d0f-b8a1-6df4f9d84c30 None admin] PATH_INFO: `/v3/domains` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:30.684175 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70b04179-1148-4d0f-b8a1-6df4f9d84c30 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:02:30.684488 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70b04179-1148-4d0f-b8a1-6df4f9d84c30 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:02:30.686029 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70b04179-1148-4d0f-b8a1-6df4f9d84c30 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:02:30.702684 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 599/1211] 10.222.0.77 () {56 vars in 1109 bytes} [Tue Aug 30 05:02:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 05:02:30.709569 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f39fb155-1d57-4af1-b2f8-d01b2f1bd770 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:02:30.820697 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f39fb155-1d57-4af1-b2f8-d01b2f1bd770 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:02:30.822435 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:02:30.863351 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9003003-b17b-4129-bff2-d7c2f170afe6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:02:30.864402 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9003003-b17b-4129-bff2-d7c2f170afe6 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:30.865180 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9003003-b17b-4129-bff2-d7c2f170afe6 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:30.865702 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9003003-b17b-4129-bff2-d7c2f170afe6 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:30.875308 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9003003-b17b-4129-bff2-d7c2f170afe6 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:02:30.877346 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9003003-b17b-4129-bff2-d7c2f170afe6 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:02:30.890560 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-b9003003-b17b-4129-bff2-d7c2f170afe6 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Aug 30 05:02:30.890560 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:02:30.890560 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:02:30.890560 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:02:30.890560 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:02:30.890560 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:02:30.890560 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:02:30.890560 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:02:30.890560 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:02:30.890560 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:02:30.890560 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:02:30.890560 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:02:30.890560 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 05:02:30.890560 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 05:02:30.890560 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 05:02:30.890560 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 05:02:30.891827 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:02:30.891827 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:02:30.891827 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 05:02:30.891827 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 05:02:30.891827 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 05:02:30.891827 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 05:02:30.891827 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:02:30.891827 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:02:30.891827 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:02:30.891827 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:02:30.891827 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:02:30.891827 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 05:02:30.891827 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:02:30.891827 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:02:30.891827 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:02:30.891827 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:02:30.891827 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:02:30.891827 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:02:30.892777 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:02:30.892777 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:02:30.892777 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 05:02:30.892777 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 05:02:30.892777 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 05:02:30.892777 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 05:02:30.892777 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 05:02:30.892777 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 05:02:30.892777 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Aug 30 05:02:30.892777 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:02:30.897553 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 613/1212] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 05:02:30 2022] GET /identity/v3/users/placement => generated 87 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:02:30.899226 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70b04179-1148-4d0f-b8a1-6df4f9d84c30 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:02:31.010978 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70b04179-1148-4d0f-b8a1-6df4f9d84c30 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:02:31.048815 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccc57d0a-7d26-4848-b6c1-3c58355704b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:02:31.049416 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccc57d0a-7d26-4848-b6c1-3c58355704b8 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:31.049699 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccc57d0a-7d26-4848-b6c1-3c58355704b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:31.049974 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccc57d0a-7d26-4848-b6c1-3c58355704b8 None admin] PATH_INFO: `/v3/users` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:31.050733 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccc57d0a-7d26-4848-b6c1-3c58355704b8 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:02:31.051054 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccc57d0a-7d26-4848-b6c1-3c58355704b8 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:02:31.052999 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccc57d0a-7d26-4848-b6c1-3c58355704b8 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:02:31.072232 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 600/1213] 10.222.0.77 () {56 vars in 1143 bytes} [Tue Aug 30 05:02:30 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:02:31.078613 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9003003-b17b-4129-bff2-d7c2f170afe6 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:02:31.201590 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9003003-b17b-4129-bff2-d7c2f170afe6 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:02:31.243846 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-321dfef7-83ce-4a78-a32d-c6f9b61dd3d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:02:31.244462 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-321dfef7-83ce-4a78-a32d-c6f9b61dd3d5 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:31.244877 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-321dfef7-83ce-4a78-a32d-c6f9b61dd3d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:31.245823 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-321dfef7-83ce-4a78-a32d-c6f9b61dd3d5 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:31.280092 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:02:31.327428 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-321dfef7-83ce-4a78-a32d-c6f9b61dd3d5 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:02:31.328944 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-321dfef7-83ce-4a78-a32d-c6f9b61dd3d5 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:02:31.442051 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 614/1214] 10.222.0.77 () {58 vars in 1282 bytes} [Tue Aug 30 05:02:31 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 364 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 30 05:02:31.448049 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccc57d0a-7d26-4848-b6c1-3c58355704b8 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:02:31.473576 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:02:31.564109 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccc57d0a-7d26-4848-b6c1-3c58355704b8 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:02:31.601198 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df5747fe-1d7d-4f78-9d9a-6e5c5be9cc53 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:02:31.602402 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df5747fe-1d7d-4f78-9d9a-6e5c5be9cc53 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:31.602896 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df5747fe-1d7d-4f78-9d9a-6e5c5be9cc53 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:31.603321 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df5747fe-1d7d-4f78-9d9a-6e5c5be9cc53 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:31.614383 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df5747fe-1d7d-4f78-9d9a-6e5c5be9cc53 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:02:31.616123 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df5747fe-1d7d-4f78-9d9a-6e5c5be9cc53 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:02:31.631412 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-df5747fe-1d7d-4f78-9d9a-6e5c5be9cc53 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 05:02:31.631412 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:02:31.631412 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:02:31.631412 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:02:31.631412 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:02:31.631412 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:02:31.631412 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:02:31.631412 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:02:31.631412 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:02:31.631412 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:02:31.631412 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:02:31.631412 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:02:31.631412 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 05:02:31.631412 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 05:02:31.631412 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 05:02:31.631412 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 05:02:31.632452 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:02:31.632452 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:02:31.632452 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:02:31.632452 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:02:31.632452 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:02:31.632452 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:02:31.632452 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:02:31.632452 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 05:02:31.632452 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:02:31.632452 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:02:31.632452 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:02:31.632452 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:02:31.632452 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:02:31.632452 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:02:31.632452 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:02:31.632452 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:02:31.632452 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 05:02:31.638671 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 05:02:31.638671 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 05:02:31.638671 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:02:31.638671 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 601/1215] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 05:02:31 2022] GET /identity/v3/domains/Default => generated 87 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:02:31.639982 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-321dfef7-83ce-4a78-a32d-c6f9b61dd3d5 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:02:31.714222 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:02:31.764570 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-321dfef7-83ce-4a78-a32d-c6f9b61dd3d5 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:02:31.804106 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c24b3af5-8121-482f-8917-323606b0c95a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:02:31.804710 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c24b3af5-8121-482f-8917-323606b0c95a None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:31.804987 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c24b3af5-8121-482f-8917-323606b0c95a None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:31.805255 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c24b3af5-8121-482f-8917-323606b0c95a None admin] PATH_INFO: `/v3/domains` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:31.805887 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c24b3af5-8121-482f-8917-323606b0c95a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:02:31.806260 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c24b3af5-8121-482f-8917-323606b0c95a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:02:31.808037 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c24b3af5-8121-482f-8917-323606b0c95a None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:02:31.826215 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 615/1216] 10.222.0.77 () {56 vars in 1109 bytes} [Tue Aug 30 05:02:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:02:31.827362 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df5747fe-1d7d-4f78-9d9a-6e5c5be9cc53 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:02:31.908657 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:02:31.951300 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df5747fe-1d7d-4f78-9d9a-6e5c5be9cc53 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:02:31.991288 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b073396-3e90-41d3-b9ab-59e8d054bd4c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:02:31.992010 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b073396-3e90-41d3-b9ab-59e8d054bd4c None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:31.992326 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b073396-3e90-41d3-b9ab-59e8d054bd4c None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:31.992602 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b073396-3e90-41d3-b9ab-59e8d054bd4c None admin] PATH_INFO: `/v3/projects/service` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:32.000563 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b073396-3e90-41d3-b9ab-59e8d054bd4c None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:02:32.002150 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b073396-3e90-41d3-b9ab-59e8d054bd4c None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:02:32.016150 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-4b073396-3e90-41d3-b9ab-59e8d054bd4c None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 05:02:32.016150 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:02:32.016150 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:02:32.016150 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:02:32.016150 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:02:32.016150 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:02:32.016150 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:02:32.016150 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:02:32.016150 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:02:32.016150 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:02:32.016150 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:02:32.016150 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:02:32.016150 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 05:02:32.016150 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 05:02:32.016150 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 05:02:32.022677 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 05:02:32.022677 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:02:32.022677 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:02:32.022677 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:02:32.022677 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:02:32.022677 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:02:32.022677 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:02:32.022677 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:02:32.022677 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 05:02:32.022677 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:02:32.022677 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:02:32.022677 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:02:32.022677 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:02:32.022677 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:02:32.022677 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:02:32.022677 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:02:32.022677 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:02:32.023974 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 05:02:32.023974 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 05:02:32.023974 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 05:02:32.023974 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 05:02:32.023974 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 05:02:32.023974 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 05:02:32.023974 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 05:02:32.023974 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:02:32.023974 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 602/1217] 10.222.0.77 () {56 vars in 1111 bytes} [Tue Aug 30 05:02:31 2022] GET /identity/v3/projects/service => generated 88 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:02:32.024758 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c24b3af5-8121-482f-8917-323606b0c95a None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:02:32.143798 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c24b3af5-8121-482f-8917-323606b0c95a None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:02:32.168208 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:02:32.184512 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9106603d-e554-46d5-b38f-24d1bc1a6380 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:02:32.185142 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9106603d-e554-46d5-b38f-24d1bc1a6380 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:32.185419 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9106603d-e554-46d5-b38f-24d1bc1a6380 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:32.185686 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9106603d-e554-46d5-b38f-24d1bc1a6380 None admin] PATH_INFO: `/v3/projects` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:32.186367 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9106603d-e554-46d5-b38f-24d1bc1a6380 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:02:32.186667 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9106603d-e554-46d5-b38f-24d1bc1a6380 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:02:32.188264 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9106603d-e554-46d5-b38f-24d1bc1a6380 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:02:32.202079 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 616/1218] 10.222.0.77 () {56 vars in 1148 bytes} [Tue Aug 30 05:02:32 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:02:32.208136 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b073396-3e90-41d3-b9ab-59e8d054bd4c None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:02:32.319283 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b073396-3e90-41d3-b9ab-59e8d054bd4c None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:02:32.353424 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:02:32.360610 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-184b2882-312b-48cf-8ce5-f6cf9b1bb336 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:02:32.361351 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-184b2882-312b-48cf-8ce5-f6cf9b1bb336 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:32.361720 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-184b2882-312b-48cf-8ce5-f6cf9b1bb336 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:32.362082 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-184b2882-312b-48cf-8ce5-f6cf9b1bb336 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:32.362977 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-184b2882-312b-48cf-8ce5-f6cf9b1bb336 None admin] RBAC: Adding query filter params (role.id=821647e8b40542499667e29e1b301cb2, scope.project.id=e9919141e3b947ab8d91b55bba4a7de6, user.id=1972959402704b89925001cd8d90cb0e) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:02:32.363391 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-184b2882-312b-48cf-8ce5-f6cf9b1bb336 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:02:32.365553 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-184b2882-312b-48cf-8ce5-f6cf9b1bb336 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:02:32.382851 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 603/1219] 10.222.0.77 () {56 vars in 1374 bytes} [Tue Aug 30 05:02:32 2022] GET /identity/v3/role_assignments?user.id=1972959402704b89925001cd8d90cb0e&scope.project.id=e9919141e3b947ab8d91b55bba4a7de6&role.id=821647e8b40542499667e29e1b301cb2 => generated 259 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:02:34.297560 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82245de3-4d76-4146-9cda-57ea37a69088 None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:34.298111 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82245de3-4d76-4146-9cda-57ea37a69088 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:34.298111 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82245de3-4d76-4146-9cda-57ea37a69088 None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:34.299457 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 617/1220] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:02:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:02:34.305114 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd23eda-3d70-4745-83a4-09c736c8b18d None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:34.305606 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd23eda-3d70-4745-83a4-09c736c8b18d None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:34.305705 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd23eda-3d70-4745-83a4-09c736c8b18d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:34.324040 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-4cd23eda-3d70-4745-83a4-09c736c8b18d None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:02:34.325217 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-4cd23eda-3d70-4745-83a4-09c736c8b18d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:02:34.399106 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 604/1221] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:02:34 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 05:02:34.678925 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82245de3-4d76-4146-9cda-57ea37a69088 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:02:34.786072 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82245de3-4d76-4146-9cda-57ea37a69088 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:02:34.820053 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-894f1738-da10-45f4-a62f-3251f5c64e2e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:02:34.820670 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-894f1738-da10-45f4-a62f-3251f5c64e2e None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:34.820940 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-894f1738-da10-45f4-a62f-3251f5c64e2e None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:34.821204 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-894f1738-da10-45f4-a62f-3251f5c64e2e None admin] PATH_INFO: `/v3/roles/service` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:34.831599 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-894f1738-da10-45f4-a62f-3251f5c64e2e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 05:02:34.836892 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-894f1738-da10-45f4-a62f-3251f5c64e2e None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 30 05:02:34.836892 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:02:34.836892 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:02:34.836892 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:02:34.836892 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:02:34.836892 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:02:34.836892 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:02:34.836892 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:02:34.836892 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:02:34.836892 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:02:34.836892 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:02:34.836892 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:02:34.836892 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 05:02:34.836892 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 05:02:34.836892 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 05:02:34.836892 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 05:02:34.837891 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 05:02:34.837891 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 05:02:34.837891 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 05:02:34.837891 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 05:02:34.837891 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:02:34.837891 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:02:34.837891 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:02:34.837891 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:02:34.837891 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:02:34.837891 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:02:34.837891 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:02:34.837891 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 05:02:34.837891 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:02:34.837891 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:02:34.837891 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:02:34.837891 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:02:34.838753 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:02:34.838753 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:02:34.838753 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:02:34.838753 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:02:34.838753 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 05:02:34.838753 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 05:02:34.838753 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 05:02:34.838753 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 05:02:34.838753 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 05:02:34.838753 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 05:02:34.838753 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 30 05:02:34.838753 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:02:34.838753 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 618/1222] 10.222.0.77 () {56 vars in 1102 bytes} [Tue Aug 30 05:02:34 2022] GET /identity/v3/roles/service => generated 85 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 05:02:34.844125 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cd23eda-3d70-4745-83a4-09c736c8b18d None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:02:34.953473 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cd23eda-3d70-4745-83a4-09c736c8b18d None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:02:34.978972 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:02:34.990169 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e5fcf77-e37d-4469-9373-0f309c0180e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:02:34.990799 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e5fcf77-e37d-4469-9373-0f309c0180e0 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:34.991153 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e5fcf77-e37d-4469-9373-0f309c0180e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:34.991425 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e5fcf77-e37d-4469-9373-0f309c0180e0 None admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:34.992078 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e5fcf77-e37d-4469-9373-0f309c0180e0 None admin] RBAC: Adding query filter params (name=service) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:02:34.992413 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e5fcf77-e37d-4469-9373-0f309c0180e0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:02:34.993910 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e5fcf77-e37d-4469-9373-0f309c0180e0 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:02:35.002408 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 605/1223] 10.222.0.77 () {56 vars in 1103 bytes} [Tue Aug 30 05:02:34 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:02:35.008887 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-894f1738-da10-45f4-a62f-3251f5c64e2e None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:02:35.055557 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:02:35.118247 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-894f1738-da10-45f4-a62f-3251f5c64e2e None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:02:35.156259 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12fe265e-efda-43c0-8720-468148452936 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:02:35.156969 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12fe265e-efda-43c0-8720-468148452936 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:35.157432 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12fe265e-efda-43c0-8720-468148452936 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:35.157715 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12fe265e-efda-43c0-8720-468148452936 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:35.170222 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12fe265e-efda-43c0-8720-468148452936 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:02:35.171991 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12fe265e-efda-43c0-8720-468148452936 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:02:35.187730 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-12fe265e-efda-43c0-8720-468148452936 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 05:02:35.187730 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:02:35.187730 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:02:35.187730 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:02:35.187730 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:02:35.187730 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:02:35.187730 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:02:35.187730 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:02:35.187730 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:02:35.187730 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:02:35.187730 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:02:35.187730 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:02:35.187730 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 05:02:35.187730 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 05:02:35.187730 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 05:02:35.187730 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 05:02:35.189020 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:02:35.189020 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:02:35.189020 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:02:35.189020 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:02:35.189020 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:02:35.189020 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:02:35.189020 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:02:35.189020 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 05:02:35.189020 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:02:35.189020 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:02:35.189020 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:02:35.189020 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:02:35.189020 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:02:35.189020 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:02:35.189020 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:02:35.189020 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:02:35.189020 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 05:02:35.190485 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 05:02:35.190485 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 05:02:35.190485 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:02:35.194845 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 619/1224] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 05:02:35 2022] GET /identity/v3/domains/Default => generated 87 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:02:35.195278 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e5fcf77-e37d-4469-9373-0f309c0180e0 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:02:35.303491 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e5fcf77-e37d-4469-9373-0f309c0180e0 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:02:35.340887 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60f615b6-196b-488b-9460-960ba45adfdf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:02:35.341523 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60f615b6-196b-488b-9460-960ba45adfdf None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:35.341757 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60f615b6-196b-488b-9460-960ba45adfdf None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:35.342059 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60f615b6-196b-488b-9460-960ba45adfdf None admin] PATH_INFO: `/v3/domains` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:35.342790 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60f615b6-196b-488b-9460-960ba45adfdf None admin] RBAC: Adding query filter params (name=Default) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:02:35.343115 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60f615b6-196b-488b-9460-960ba45adfdf None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:02:35.344689 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60f615b6-196b-488b-9460-960ba45adfdf None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:02:35.358224 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 606/1225] 10.222.0.77 () {56 vars in 1109 bytes} [Tue Aug 30 05:02:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:02:35.364114 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12fe265e-efda-43c0-8720-468148452936 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:02:35.467001 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:02:35.474656 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12fe265e-efda-43c0-8720-468148452936 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:02:35.511471 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63db236e-78dd-49e2-b98c-98a48494965b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:02:35.512078 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63db236e-78dd-49e2-b98c-98a48494965b None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:35.512359 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63db236e-78dd-49e2-b98c-98a48494965b None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:35.512624 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63db236e-78dd-49e2-b98c-98a48494965b None admin] PATH_INFO: `/v3/users/placement` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:35.520872 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63db236e-78dd-49e2-b98c-98a48494965b None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:02:35.522311 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63db236e-78dd-49e2-b98c-98a48494965b None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:02:35.535699 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-63db236e-78dd-49e2-b98c-98a48494965b None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Aug 30 05:02:35.535699 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:02:35.535699 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:02:35.535699 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:02:35.535699 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:02:35.535699 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:02:35.535699 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:02:35.535699 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:02:35.535699 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:02:35.535699 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:02:35.535699 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:02:35.535699 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:02:35.535699 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 05:02:35.535699 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 05:02:35.535699 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 05:02:35.535699 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 05:02:35.536829 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:02:35.536829 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:02:35.536829 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 05:02:35.536829 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 05:02:35.536829 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 05:02:35.536829 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 05:02:35.536829 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:02:35.536829 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:02:35.536829 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:02:35.536829 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:02:35.536829 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:02:35.536829 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 05:02:35.536829 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:02:35.536829 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:02:35.536829 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:02:35.536829 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:02:35.536829 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:02:35.536829 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:02:35.538159 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:02:35.538159 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:02:35.538159 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 05:02:35.538159 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 05:02:35.538159 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 05:02:35.538159 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 05:02:35.538159 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 05:02:35.538159 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 05:02:35.538159 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Aug 30 05:02:35.538159 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:02:35.538159 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 620/1226] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 05:02:35 2022] GET /identity/v3/users/placement => generated 87 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:02:35.542796 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60f615b6-196b-488b-9460-960ba45adfdf None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:02:35.595895 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:02:35.661972 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60f615b6-196b-488b-9460-960ba45adfdf None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:02:35.700334 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e97a364-c78a-440b-aeff-e130c0898935 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:02:35.701016 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e97a364-c78a-440b-aeff-e130c0898935 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:35.701328 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e97a364-c78a-440b-aeff-e130c0898935 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:35.701664 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e97a364-c78a-440b-aeff-e130c0898935 None admin] PATH_INFO: `/v3/users` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:35.702539 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e97a364-c78a-440b-aeff-e130c0898935 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:02:35.702860 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e97a364-c78a-440b-aeff-e130c0898935 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:02:35.704442 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e97a364-c78a-440b-aeff-e130c0898935 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:02:35.723532 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 607/1227] 10.222.0.77 () {56 vars in 1143 bytes} [Tue Aug 30 05:02:35 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 05:02:35.730236 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63db236e-78dd-49e2-b98c-98a48494965b None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:02:35.840456 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63db236e-78dd-49e2-b98c-98a48494965b None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:02:35.878064 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48b57a91-d353-42c0-ae56-0fe0f5358006 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:02:35.878695 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48b57a91-d353-42c0-ae56-0fe0f5358006 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:35.879107 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48b57a91-d353-42c0-ae56-0fe0f5358006 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:35.879358 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48b57a91-d353-42c0-ae56-0fe0f5358006 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:35.891443 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48b57a91-d353-42c0-ae56-0fe0f5358006 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:02:35.893120 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48b57a91-d353-42c0-ae56-0fe0f5358006 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:02:35.909474 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-48b57a91-d353-42c0-ae56-0fe0f5358006 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 05:02:35.909474 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:02:35.909474 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:02:35.909474 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:02:35.909474 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:02:35.909474 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:02:35.909474 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:02:35.909474 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:02:35.909474 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:02:35.909474 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:02:35.909474 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:02:35.909474 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:02:35.909474 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 05:02:35.909474 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 05:02:35.909474 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 05:02:35.909474 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 05:02:35.910642 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:02:35.910642 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:02:35.910642 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:02:35.910642 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:02:35.910642 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:02:35.910642 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:02:35.910642 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:02:35.910642 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 05:02:35.910642 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:02:35.910642 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:02:35.910642 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:02:35.910642 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:02:35.910642 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:02:35.910642 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:02:35.910642 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:02:35.910642 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:02:35.910642 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 05:02:35.912220 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 05:02:35.912220 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 05:02:35.912220 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:02:35.912220 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 621/1228] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 05:02:35 2022] GET /identity/v3/domains/Default => generated 87 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 05:02:35.918018 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e97a364-c78a-440b-aeff-e130c0898935 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:02:36.038662 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e97a364-c78a-440b-aeff-e130c0898935 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:02:36.041552 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:02:36.076897 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-523f8eb2-34a9-45ad-a9b1-e88b99bb725d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:02:36.077658 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-523f8eb2-34a9-45ad-a9b1-e88b99bb725d None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:36.077960 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-523f8eb2-34a9-45ad-a9b1-e88b99bb725d None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:36.078271 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-523f8eb2-34a9-45ad-a9b1-e88b99bb725d None admin] PATH_INFO: `/v3/domains` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:36.079287 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-523f8eb2-34a9-45ad-a9b1-e88b99bb725d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:02:36.079715 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-523f8eb2-34a9-45ad-a9b1-e88b99bb725d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:02:36.081193 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-523f8eb2-34a9-45ad-a9b1-e88b99bb725d None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:02:36.093996 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 608/1229] 10.222.0.77 () {56 vars in 1109 bytes} [Tue Aug 30 05:02:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:02:36.101268 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48b57a91-d353-42c0-ae56-0fe0f5358006 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:02:36.111955 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:02:36.215134 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48b57a91-d353-42c0-ae56-0fe0f5358006 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:02:36.250794 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa71954d-918a-4f73-96ea-90ffdf16cbaa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:02:36.251448 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa71954d-918a-4f73-96ea-90ffdf16cbaa None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:36.251721 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa71954d-918a-4f73-96ea-90ffdf16cbaa None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:36.251988 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa71954d-918a-4f73-96ea-90ffdf16cbaa None admin] PATH_INFO: `/v3/projects/service` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:36.259213 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa71954d-918a-4f73-96ea-90ffdf16cbaa None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:02:36.260778 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa71954d-918a-4f73-96ea-90ffdf16cbaa None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:02:36.272548 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-aa71954d-918a-4f73-96ea-90ffdf16cbaa None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 05:02:36.272548 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:02:36.272548 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:02:36.272548 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:02:36.272548 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:02:36.272548 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:02:36.272548 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:02:36.272548 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:02:36.272548 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:02:36.272548 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:02:36.272548 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:02:36.272548 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:02:36.272548 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 05:02:36.272548 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 05:02:36.272548 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 05:02:36.273603 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 05:02:36.273603 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:02:36.273603 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:02:36.273603 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:02:36.273603 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:02:36.273603 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:02:36.273603 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:02:36.273603 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:02:36.273603 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 05:02:36.273603 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:02:36.273603 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:02:36.273603 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:02:36.273603 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:02:36.273603 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:02:36.273603 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:02:36.273603 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:02:36.273603 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:02:36.274509 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 05:02:36.274509 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 05:02:36.274509 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 05:02:36.274509 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 05:02:36.274509 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 05:02:36.274509 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 05:02:36.274509 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 05:02:36.274509 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:02:36.274509 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 622/1230] 10.222.0.77 () {56 vars in 1111 bytes} [Tue Aug 30 05:02:36 2022] GET /identity/v3/projects/service => generated 88 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:02:36.279442 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-523f8eb2-34a9-45ad-a9b1-e88b99bb725d None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:02:36.393775 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-523f8eb2-34a9-45ad-a9b1-e88b99bb725d None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:02:36.431957 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99d19a34-0ca7-43d8-8179-f2a71611dea9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:02:36.432507 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99d19a34-0ca7-43d8-8179-f2a71611dea9 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:36.432781 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99d19a34-0ca7-43d8-8179-f2a71611dea9 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:36.433049 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99d19a34-0ca7-43d8-8179-f2a71611dea9 None admin] PATH_INFO: `/v3/projects` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:36.433723 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99d19a34-0ca7-43d8-8179-f2a71611dea9 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:02:36.434040 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99d19a34-0ca7-43d8-8179-f2a71611dea9 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:02:36.435521 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99d19a34-0ca7-43d8-8179-f2a71611dea9 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:02:36.448698 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 609/1231] 10.222.0.77 () {56 vars in 1148 bytes} [Tue Aug 30 05:02:36 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:02:36.455047 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa71954d-918a-4f73-96ea-90ffdf16cbaa None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:02:36.525838 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:02:36.568348 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa71954d-918a-4f73-96ea-90ffdf16cbaa None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:02:36.606306 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a1c8f77-e261-40bc-8bf4-eb86e9ccfe3f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:02:36.606984 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a1c8f77-e261-40bc-8bf4-eb86e9ccfe3f None admin] REQUEST_METHOD: `PUT` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:36.607260 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a1c8f77-e261-40bc-8bf4-eb86e9ccfe3f None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:36.607527 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a1c8f77-e261-40bc-8bf4-eb86e9ccfe3f None admin] PATH_INFO: `/v3/projects/e9919141e3b947ab8d91b55bba4a7de6/users/1972959402704b89925001cd8d90cb0e/roles/821647e8b40542499667e29e1b301cb2` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:36.622994 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a1c8f77-e261-40bc-8bf4-eb86e9ccfe3f None admin] RBAC: Authorizing `identity:create_grant(project_id=e9919141e3b947ab8d91b55bba4a7de6, user_id=1972959402704b89925001cd8d90cb0e, role_id=821647e8b40542499667e29e1b301cb2)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:02:36.624418 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a1c8f77-e261-40bc-8bf4-eb86e9ccfe3f None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:02:36.640997 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 623/1232] 10.222.0.77 () {58 vars in 1439 bytes} [Tue Aug 30 05:02:36 2022] PUT /identity/v3/projects/e9919141e3b947ab8d91b55bba4a7de6/users/1972959402704b89925001cd8d90cb0e/roles/821647e8b40542499667e29e1b301cb2 => generated 0 bytes in 186 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 05:02:38.743695 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64d16aff-2a7c-4961-aa12-70d6ac5d2ed1 None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:38.744250 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64d16aff-2a7c-4961-aa12-70d6ac5d2ed1 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:38.744250 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64d16aff-2a7c-4961-aa12-70d6ac5d2ed1 None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:38.745646 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 610/1233] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:02:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:02:38.751758 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa77564c-4467-4e5d-9eb0-19077cf19afc None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:38.752204 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa77564c-4467-4e5d-9eb0-19077cf19afc None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:38.752396 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa77564c-4467-4e5d-9eb0-19077cf19afc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:38.772706 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-aa77564c-4467-4e5d-9eb0-19077cf19afc None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:02:38.774097 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-aa77564c-4467-4e5d-9eb0-19077cf19afc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:02:38.843720 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 624/1234] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:02:38 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 05:02:39.137405 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64d16aff-2a7c-4961-aa12-70d6ac5d2ed1 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:02:39.167294 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:02:39.255510 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64d16aff-2a7c-4961-aa12-70d6ac5d2ed1 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:02:39.291376 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1d82f26-037a-4fcf-9142-15033e365f83 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:02:39.292000 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d82f26-037a-4fcf-9142-15033e365f83 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:39.292270 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d82f26-037a-4fcf-9142-15033e365f83 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:39.292619 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d82f26-037a-4fcf-9142-15033e365f83 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:39.303035 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1d82f26-037a-4fcf-9142-15033e365f83 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 05:02:39.308744 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-d1d82f26-037a-4fcf-9142-15033e365f83 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 30 05:02:39.308744 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:02:39.308744 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:02:39.308744 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:02:39.308744 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:02:39.308744 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:02:39.308744 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:02:39.308744 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:02:39.308744 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:02:39.308744 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:02:39.308744 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:02:39.308744 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:02:39.308744 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 05:02:39.308744 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 05:02:39.308744 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 05:02:39.308744 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 05:02:39.310005 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 05:02:39.310005 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 05:02:39.310005 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 05:02:39.310005 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 05:02:39.310005 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:02:39.310005 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:02:39.310005 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:02:39.310005 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:02:39.310005 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:02:39.310005 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:02:39.310005 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:02:39.310005 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 05:02:39.310005 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:02:39.310005 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:02:39.310005 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:02:39.310005 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:02:39.310907 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:02:39.310907 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:02:39.310907 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:02:39.310907 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:02:39.310907 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 05:02:39.310907 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 05:02:39.310907 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 05:02:39.310907 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 05:02:39.310907 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 05:02:39.310907 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 05:02:39.310907 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 30 05:02:39.310907 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:02:39.310907 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 611/1235] 10.222.0.77 () {56 vars in 1102 bytes} [Tue Aug 30 05:02:39 2022] GET /identity/v3/roles/service => generated 85 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:02:39.316381 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa77564c-4467-4e5d-9eb0-19077cf19afc None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:02:39.367781 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:02:39.425053 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa77564c-4467-4e5d-9eb0-19077cf19afc None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:02:39.463087 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-373c7edf-6c90-4474-91d0-2e0e7d10799e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:02:39.464035 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373c7edf-6c90-4474-91d0-2e0e7d10799e None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:39.464518 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373c7edf-6c90-4474-91d0-2e0e7d10799e None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:39.464801 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373c7edf-6c90-4474-91d0-2e0e7d10799e None admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:39.465852 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-373c7edf-6c90-4474-91d0-2e0e7d10799e None admin] RBAC: Adding query filter params (name=service) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:02:39.465852 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-373c7edf-6c90-4474-91d0-2e0e7d10799e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:02:39.467150 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-373c7edf-6c90-4474-91d0-2e0e7d10799e None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:02:39.475948 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 625/1236] 10.222.0.77 () {56 vars in 1103 bytes} [Tue Aug 30 05:02:39 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:02:39.481973 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1d82f26-037a-4fcf-9142-15033e365f83 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:02:39.573371 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:02:39.590699 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1d82f26-037a-4fcf-9142-15033e365f83 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:02:39.626196 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26dad4e2-0ca2-4235-acb1-40e68958e4f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:02:39.626788 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26dad4e2-0ca2-4235-acb1-40e68958e4f8 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:39.627096 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26dad4e2-0ca2-4235-acb1-40e68958e4f8 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:39.627368 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26dad4e2-0ca2-4235-acb1-40e68958e4f8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:39.640340 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26dad4e2-0ca2-4235-acb1-40e68958e4f8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:02:39.641989 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26dad4e2-0ca2-4235-acb1-40e68958e4f8 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:02:39.658125 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-26dad4e2-0ca2-4235-acb1-40e68958e4f8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 05:02:39.658125 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:02:39.658125 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:02:39.658125 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:02:39.658125 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:02:39.658125 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:02:39.658125 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:02:39.658125 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:02:39.658125 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:02:39.658125 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:02:39.658125 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:02:39.658125 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:02:39.658125 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 05:02:39.658125 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 05:02:39.658125 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 05:02:39.658125 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 05:02:39.659264 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:02:39.659264 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:02:39.659264 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:02:39.659264 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:02:39.659264 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:02:39.659264 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:02:39.659264 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:02:39.659264 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 05:02:39.659264 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:02:39.659264 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:02:39.659264 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:02:39.659264 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:02:39.659264 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:02:39.659264 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:02:39.659264 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:02:39.659264 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:02:39.659264 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 05:02:39.660163 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 05:02:39.660163 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 05:02:39.660163 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:02:39.660163 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 612/1237] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 05:02:39 2022] GET /identity/v3/domains/Default => generated 87 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:02:39.665590 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-373c7edf-6c90-4474-91d0-2e0e7d10799e None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:02:39.776382 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-373c7edf-6c90-4474-91d0-2e0e7d10799e None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:02:39.786397 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:02:39.813346 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14dd8732-b44d-4a65-943d-44c32366fae2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:02:39.815005 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14dd8732-b44d-4a65-943d-44c32366fae2 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:39.815005 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14dd8732-b44d-4a65-943d-44c32366fae2 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:39.815005 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14dd8732-b44d-4a65-943d-44c32366fae2 None admin] PATH_INFO: `/v3/domains` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:39.815270 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14dd8732-b44d-4a65-943d-44c32366fae2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:02:39.815399 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14dd8732-b44d-4a65-943d-44c32366fae2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:02:39.816748 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14dd8732-b44d-4a65-943d-44c32366fae2 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:02:39.829805 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 626/1238] 10.222.0.77 () {56 vars in 1109 bytes} [Tue Aug 30 05:02:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:02:39.835682 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26dad4e2-0ca2-4235-acb1-40e68958e4f8 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:02:39.948503 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26dad4e2-0ca2-4235-acb1-40e68958e4f8 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:02:39.991124 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43c23e3e-cc1c-4b46-9870-1659155af793 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:02:39.991756 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43c23e3e-cc1c-4b46-9870-1659155af793 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:39.992038 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43c23e3e-cc1c-4b46-9870-1659155af793 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:39.992318 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43c23e3e-cc1c-4b46-9870-1659155af793 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:39.995369 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:02:40.001540 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43c23e3e-cc1c-4b46-9870-1659155af793 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:02:40.003285 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43c23e3e-cc1c-4b46-9870-1659155af793 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:02:40.016557 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-43c23e3e-cc1c-4b46-9870-1659155af793 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Aug 30 05:02:40.016557 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:02:40.016557 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:02:40.016557 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:02:40.016557 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:02:40.016557 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:02:40.016557 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:02:40.016557 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:02:40.016557 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:02:40.016557 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:02:40.016557 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:02:40.016557 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:02:40.016557 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 05:02:40.016557 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 05:02:40.016557 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 05:02:40.016557 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 05:02:40.017621 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:02:40.017621 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:02:40.017621 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 05:02:40.017621 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 05:02:40.017621 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 05:02:40.017621 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 05:02:40.017621 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:02:40.017621 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:02:40.017621 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:02:40.017621 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:02:40.017621 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:02:40.017621 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 05:02:40.017621 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:02:40.017621 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:02:40.017621 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:02:40.017621 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:02:40.017621 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:02:40.017621 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:02:40.018598 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:02:40.018598 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:02:40.018598 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 05:02:40.018598 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 05:02:40.018598 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 05:02:40.018598 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 05:02:40.018598 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 05:02:40.018598 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 05:02:40.018598 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Aug 30 05:02:40.018598 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:02:40.018598 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 613/1239] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 05:02:39 2022] GET /identity/v3/users/placement => generated 87 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 05:02:40.024520 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14dd8732-b44d-4a65-943d-44c32366fae2 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:02:40.134981 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14dd8732-b44d-4a65-943d-44c32366fae2 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:02:40.171175 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69f9cac2-3fec-4946-b448-d77ec1c3bd06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:02:40.171782 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69f9cac2-3fec-4946-b448-d77ec1c3bd06 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:40.172117 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69f9cac2-3fec-4946-b448-d77ec1c3bd06 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:40.172333 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69f9cac2-3fec-4946-b448-d77ec1c3bd06 None admin] PATH_INFO: `/v3/users` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:40.173038 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69f9cac2-3fec-4946-b448-d77ec1c3bd06 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:02:40.173337 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69f9cac2-3fec-4946-b448-d77ec1c3bd06 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:02:40.174768 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69f9cac2-3fec-4946-b448-d77ec1c3bd06 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:02:40.193286 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 627/1240] 10.222.0.77 () {56 vars in 1143 bytes} [Tue Aug 30 05:02:40 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:02:40.199126 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43c23e3e-cc1c-4b46-9870-1659155af793 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:02:40.300946 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43c23e3e-cc1c-4b46-9870-1659155af793 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:02:40.339742 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2d244da-e5d1-4df1-a943-1ea63db4cbd3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:02:40.340420 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d244da-e5d1-4df1-a943-1ea63db4cbd3 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:40.340732 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d244da-e5d1-4df1-a943-1ea63db4cbd3 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:40.341062 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d244da-e5d1-4df1-a943-1ea63db4cbd3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:40.413087 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:02:40.420469 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2d244da-e5d1-4df1-a943-1ea63db4cbd3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:02:40.422074 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2d244da-e5d1-4df1-a943-1ea63db4cbd3 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:02:40.529874 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 614/1241] 10.222.0.77 () {58 vars in 1282 bytes} [Tue Aug 30 05:02:40 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 331 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Aug 30 05:02:40.535940 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69f9cac2-3fec-4946-b448-d77ec1c3bd06 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:02:40.566959 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:02:40.653347 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69f9cac2-3fec-4946-b448-d77ec1c3bd06 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:02:40.688564 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8446edd7-ffa0-4822-ba82-eab2efba681c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:02:40.689105 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8446edd7-ffa0-4822-ba82-eab2efba681c None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:40.689376 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8446edd7-ffa0-4822-ba82-eab2efba681c None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:40.689640 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8446edd7-ffa0-4822-ba82-eab2efba681c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:40.699478 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8446edd7-ffa0-4822-ba82-eab2efba681c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:02:40.700879 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8446edd7-ffa0-4822-ba82-eab2efba681c None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:02:40.715829 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-8446edd7-ffa0-4822-ba82-eab2efba681c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 05:02:40.715829 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:02:40.715829 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:02:40.715829 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:02:40.715829 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:02:40.715829 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:02:40.715829 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:02:40.715829 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:02:40.715829 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:02:40.715829 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:02:40.715829 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:02:40.715829 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:02:40.715829 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 05:02:40.715829 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 05:02:40.715829 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 05:02:40.715829 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 05:02:40.716853 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:02:40.716853 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:02:40.716853 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:02:40.716853 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:02:40.716853 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:02:40.716853 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:02:40.716853 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:02:40.716853 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 05:02:40.716853 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:02:40.716853 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:02:40.716853 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:02:40.716853 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:02:40.716853 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:02:40.716853 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:02:40.716853 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:02:40.716853 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:02:40.716853 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 05:02:40.722530 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 05:02:40.722530 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 05:02:40.722530 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:02:40.722530 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 628/1242] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 05:02:40 2022] GET /identity/v3/domains/Default => generated 87 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:02:40.722768 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2d244da-e5d1-4df1-a943-1ea63db4cbd3 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:02:40.832490 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:02:40.841176 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2d244da-e5d1-4df1-a943-1ea63db4cbd3 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:02:40.878917 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7e4237f-5e4b-4724-90f9-312a080e83f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:02:40.879508 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7e4237f-5e4b-4724-90f9-312a080e83f4 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:40.879788 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7e4237f-5e4b-4724-90f9-312a080e83f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:40.880539 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7e4237f-5e4b-4724-90f9-312a080e83f4 None admin] PATH_INFO: `/v3/domains` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:40.881166 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7e4237f-5e4b-4724-90f9-312a080e83f4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:02:40.881460 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7e4237f-5e4b-4724-90f9-312a080e83f4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:02:40.882839 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7e4237f-5e4b-4724-90f9-312a080e83f4 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:02:40.893957 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 615/1243] 10.222.0.77 () {56 vars in 1109 bytes} [Tue Aug 30 05:02:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:02:40.900382 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8446edd7-ffa0-4822-ba82-eab2efba681c None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:02:40.992875 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:02:41.020744 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8446edd7-ffa0-4822-ba82-eab2efba681c None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:02:41.061727 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ceaee38-84a0-463a-942a-8e390887220c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:02:41.062341 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ceaee38-84a0-463a-942a-8e390887220c None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:41.062738 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ceaee38-84a0-463a-942a-8e390887220c None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:41.063037 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ceaee38-84a0-463a-942a-8e390887220c None admin] PATH_INFO: `/v3/projects/service` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:41.071817 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ceaee38-84a0-463a-942a-8e390887220c None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:02:41.073474 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ceaee38-84a0-463a-942a-8e390887220c None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:02:41.086637 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-5ceaee38-84a0-463a-942a-8e390887220c None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 05:02:41.086637 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:02:41.086637 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:02:41.086637 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:02:41.086637 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:02:41.086637 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:02:41.086637 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:02:41.086637 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:02:41.086637 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:02:41.086637 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:02:41.086637 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:02:41.086637 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:02:41.086637 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 05:02:41.086637 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 05:02:41.086637 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 05:02:41.088036 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 05:02:41.088036 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:02:41.088036 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:02:41.088036 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:02:41.088036 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:02:41.088036 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:02:41.088036 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:02:41.088036 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:02:41.088036 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 05:02:41.088036 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:02:41.088036 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:02:41.088036 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:02:41.088036 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:02:41.088036 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:02:41.088036 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:02:41.088036 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:02:41.088036 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:02:41.089409 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 05:02:41.089409 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 05:02:41.089409 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 05:02:41.089409 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 05:02:41.089409 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 05:02:41.089409 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 05:02:41.089409 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 05:02:41.089409 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:02:41.089409 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 629/1244] 10.222.0.77 () {56 vars in 1111 bytes} [Tue Aug 30 05:02:40 2022] GET /identity/v3/projects/service => generated 88 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:02:41.094052 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7e4237f-5e4b-4724-90f9-312a080e83f4 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:02:41.205586 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7e4237f-5e4b-4724-90f9-312a080e83f4 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:02:41.247983 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdd8ca81-db2e-43de-86e3-c428c249df8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:02:41.248587 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdd8ca81-db2e-43de-86e3-c428c249df8f None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:41.248901 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdd8ca81-db2e-43de-86e3-c428c249df8f None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:41.249221 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdd8ca81-db2e-43de-86e3-c428c249df8f None admin] PATH_INFO: `/v3/projects` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:41.249953 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdd8ca81-db2e-43de-86e3-c428c249df8f None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:02:41.250261 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdd8ca81-db2e-43de-86e3-c428c249df8f None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:02:41.251822 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdd8ca81-db2e-43de-86e3-c428c249df8f None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:02:41.265642 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 616/1245] 10.222.0.77 () {56 vars in 1148 bytes} [Tue Aug 30 05:02:41 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:02:41.272427 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ceaee38-84a0-463a-942a-8e390887220c None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:02:41.383887 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ceaee38-84a0-463a-942a-8e390887220c None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:02:41.419327 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8650bedd-f2e0-4547-97da-11210b58ecf5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:02:41.419963 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8650bedd-f2e0-4547-97da-11210b58ecf5 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:41.420237 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8650bedd-f2e0-4547-97da-11210b58ecf5 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:41.420528 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8650bedd-f2e0-4547-97da-11210b58ecf5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:41.421204 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8650bedd-f2e0-4547-97da-11210b58ecf5 None admin] RBAC: Adding query filter params (role.id=821647e8b40542499667e29e1b301cb2, scope.project.id=e9919141e3b947ab8d91b55bba4a7de6, user.id=1972959402704b89925001cd8d90cb0e) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:02:41.421500 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8650bedd-f2e0-4547-97da-11210b58ecf5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:02:41.423052 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8650bedd-f2e0-4547-97da-11210b58ecf5 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:02:41.430037 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 630/1246] 10.222.0.77 () {56 vars in 1374 bytes} [Tue Aug 30 05:02:41 2022] GET /identity/v3/role_assignments?user.id=1972959402704b89925001cd8d90cb0e&scope.project.id=e9919141e3b947ab8d91b55bba4a7de6&role.id=821647e8b40542499667e29e1b301cb2 => generated 608 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 05:02:43.632779 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6167209c-3017-4650-913c-4e7c786618d1 None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:43.633346 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6167209c-3017-4650-913c-4e7c786618d1 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:43.633438 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6167209c-3017-4650-913c-4e7c786618d1 None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:43.634876 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 617/1247] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:02:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:02:43.640599 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f5a902-8829-4cf0-b5af-a082bec91054 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:43.641114 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f5a902-8829-4cf0-b5af-a082bec91054 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:43.641244 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f5a902-8829-4cf0-b5af-a082bec91054 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:43.647958 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:02:43.661903 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-24f5a902-8829-4cf0-b5af-a082bec91054 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:02:43.663132 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-24f5a902-8829-4cf0-b5af-a082bec91054 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:02:43.738852 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 631/1248] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:02:43 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 05:02:44.054657 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6167209c-3017-4650-913c-4e7c786618d1 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:02:44.076205 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:02:44.173579 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6167209c-3017-4650-913c-4e7c786618d1 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:02:44.212713 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ea2bd90-6b9b-4791-8937-82750671b5e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:02:44.213347 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ea2bd90-6b9b-4791-8937-82750671b5e3 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:44.213624 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ea2bd90-6b9b-4791-8937-82750671b5e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:44.213890 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ea2bd90-6b9b-4791-8937-82750671b5e3 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:44.225014 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ea2bd90-6b9b-4791-8937-82750671b5e3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 05:02:44.230379 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-8ea2bd90-6b9b-4791-8937-82750671b5e3 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 05:02:44.230379 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:02:44.230379 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:02:44.230379 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:02:44.230379 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:02:44.230379 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:02:44.230379 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:02:44.230379 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:02:44.230379 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:02:44.230379 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:02:44.230379 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:02:44.230379 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:02:44.230379 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 05:02:44.230379 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 05:02:44.230379 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 05:02:44.230379 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 05:02:44.231708 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 05:02:44.231708 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 05:02:44.231708 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 05:02:44.231708 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 05:02:44.231708 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:02:44.231708 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:02:44.231708 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:02:44.231708 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:02:44.231708 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:02:44.231708 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:02:44.231708 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:02:44.231708 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 05:02:44.231708 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:02:44.231708 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:02:44.231708 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:02:44.231708 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:02:44.232986 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:02:44.232986 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:02:44.232986 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:02:44.232986 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:02:44.232986 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 05:02:44.232986 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 05:02:44.232986 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 05:02:44.232986 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 05:02:44.232986 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 05:02:44.232986 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 05:02:44.232986 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 05:02:44.232986 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:02:44.232986 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 618/1249] 10.222.0.77 () {56 vars in 1096 bytes} [Tue Aug 30 05:02:44 2022] GET /identity/v3/roles/admin => generated 83 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:02:44.237840 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24f5a902-8829-4cf0-b5af-a082bec91054 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:02:44.353166 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24f5a902-8829-4cf0-b5af-a082bec91054 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:02:44.386622 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:02:44.392128 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a480fb63-7b7b-40db-a45c-41c40b7a0925 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:02:44.392724 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a480fb63-7b7b-40db-a45c-41c40b7a0925 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:44.393000 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a480fb63-7b7b-40db-a45c-41c40b7a0925 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:44.393291 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a480fb63-7b7b-40db-a45c-41c40b7a0925 None admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:44.393941 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a480fb63-7b7b-40db-a45c-41c40b7a0925 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:02:44.394235 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a480fb63-7b7b-40db-a45c-41c40b7a0925 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:02:44.396113 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a480fb63-7b7b-40db-a45c-41c40b7a0925 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:02:44.405326 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 632/1250] 10.222.0.77 () {56 vars in 1099 bytes} [Tue Aug 30 05:02:44 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:02:44.412084 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ea2bd90-6b9b-4791-8937-82750671b5e3 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:02:44.496749 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:02:44.521642 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ea2bd90-6b9b-4791-8937-82750671b5e3 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:02:44.561878 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb4a3622-3417-4132-b73b-d0ffa8695b86 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:02:44.562552 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb4a3622-3417-4132-b73b-d0ffa8695b86 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:44.562905 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb4a3622-3417-4132-b73b-d0ffa8695b86 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:44.563205 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb4a3622-3417-4132-b73b-d0ffa8695b86 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:44.574356 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb4a3622-3417-4132-b73b-d0ffa8695b86 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:02:44.575933 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb4a3622-3417-4132-b73b-d0ffa8695b86 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:02:44.591156 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-fb4a3622-3417-4132-b73b-d0ffa8695b86 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 05:02:44.591156 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:02:44.591156 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:02:44.591156 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:02:44.591156 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:02:44.591156 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:02:44.591156 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:02:44.591156 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:02:44.591156 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:02:44.591156 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:02:44.591156 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:02:44.591156 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:02:44.591156 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 05:02:44.591156 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 05:02:44.591156 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 05:02:44.591156 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 05:02:44.592314 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:02:44.592314 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:02:44.592314 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:02:44.592314 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:02:44.592314 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:02:44.592314 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:02:44.592314 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:02:44.592314 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 05:02:44.592314 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:02:44.592314 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:02:44.592314 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:02:44.592314 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:02:44.592314 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:02:44.592314 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:02:44.592314 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:02:44.592314 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:02:44.592314 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 05:02:44.593475 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 05:02:44.593475 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 05:02:44.593475 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:02:44.593475 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 619/1251] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 05:02:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 05:02:44.600462 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a480fb63-7b7b-40db-a45c-41c40b7a0925 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:02:44.705878 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a480fb63-7b7b-40db-a45c-41c40b7a0925 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:02:44.743377 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b385f70e-7c80-4805-a460-d906fe0e3d8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:02:44.743974 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b385f70e-7c80-4805-a460-d906fe0e3d8d None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:44.744247 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b385f70e-7c80-4805-a460-d906fe0e3d8d None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:44.744514 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b385f70e-7c80-4805-a460-d906fe0e3d8d None admin] PATH_INFO: `/v3/domains` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:44.745140 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b385f70e-7c80-4805-a460-d906fe0e3d8d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:02:44.745435 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b385f70e-7c80-4805-a460-d906fe0e3d8d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:02:44.746795 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b385f70e-7c80-4805-a460-d906fe0e3d8d None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:02:44.759104 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 633/1252] 10.222.0.77 () {56 vars in 1109 bytes} [Tue Aug 30 05:02:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 05:02:44.764806 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb4a3622-3417-4132-b73b-d0ffa8695b86 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:02:44.868959 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb4a3622-3417-4132-b73b-d0ffa8695b86 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:02:44.898709 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:02:44.907187 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bead365-c3b0-4e84-ac0e-99bdc71a0ef3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:02:44.908047 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bead365-c3b0-4e84-ac0e-99bdc71a0ef3 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:44.908335 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bead365-c3b0-4e84-ac0e-99bdc71a0ef3 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:44.908645 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bead365-c3b0-4e84-ac0e-99bdc71a0ef3 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:44.917489 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bead365-c3b0-4e84-ac0e-99bdc71a0ef3 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:02:44.919158 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bead365-c3b0-4e84-ac0e-99bdc71a0ef3 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:02:44.932706 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-6bead365-c3b0-4e84-ac0e-99bdc71a0ef3 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Aug 30 05:02:44.932706 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:02:44.932706 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:02:44.932706 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:02:44.932706 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:02:44.932706 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:02:44.932706 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:02:44.932706 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:02:44.932706 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:02:44.932706 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:02:44.932706 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:02:44.932706 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:02:44.932706 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 05:02:44.932706 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 05:02:44.932706 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 05:02:44.932706 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 05:02:44.933913 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:02:44.933913 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:02:44.933913 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 05:02:44.933913 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 05:02:44.933913 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 05:02:44.933913 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 05:02:44.933913 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:02:44.933913 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:02:44.933913 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:02:44.933913 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:02:44.933913 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:02:44.933913 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 05:02:44.933913 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:02:44.933913 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:02:44.933913 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:02:44.933913 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:02:44.933913 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:02:44.933913 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:02:44.935058 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:02:44.935058 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:02:44.935058 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 05:02:44.935058 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 05:02:44.935058 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 05:02:44.935058 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 05:02:44.935058 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 05:02:44.935058 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 05:02:44.935058 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Aug 30 05:02:44.935058 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:02:44.935058 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 620/1253] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 05:02:44 2022] GET /identity/v3/users/placement => generated 87 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 05:02:44.940172 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b385f70e-7c80-4805-a460-d906fe0e3d8d None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:02:45.003427 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:02:45.054043 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b385f70e-7c80-4805-a460-d906fe0e3d8d None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:02:45.094092 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-387d4712-a007-4eb6-9787-49a1bb7b6570 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:02:45.094693 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-387d4712-a007-4eb6-9787-49a1bb7b6570 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:45.095006 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-387d4712-a007-4eb6-9787-49a1bb7b6570 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:45.095285 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-387d4712-a007-4eb6-9787-49a1bb7b6570 None admin] PATH_INFO: `/v3/users` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:45.096081 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-387d4712-a007-4eb6-9787-49a1bb7b6570 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:02:45.096427 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-387d4712-a007-4eb6-9787-49a1bb7b6570 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:02:45.097920 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-387d4712-a007-4eb6-9787-49a1bb7b6570 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:02:45.117765 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 634/1254] 10.222.0.77 () {56 vars in 1143 bytes} [Tue Aug 30 05:02:44 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:02:45.123666 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bead365-c3b0-4e84-ac0e-99bdc71a0ef3 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:02:45.239946 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bead365-c3b0-4e84-ac0e-99bdc71a0ef3 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:02:45.276098 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f42ef2c0-928d-4213-8d1c-027c764742a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:02:45.276617 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f42ef2c0-928d-4213-8d1c-027c764742a8 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:45.276891 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f42ef2c0-928d-4213-8d1c-027c764742a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:45.277156 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f42ef2c0-928d-4213-8d1c-027c764742a8 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:45.340849 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:02:45.357372 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f42ef2c0-928d-4213-8d1c-027c764742a8 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:02:45.358837 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f42ef2c0-928d-4213-8d1c-027c764742a8 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:02:45.477513 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 621/1255] 10.222.0.77 () {58 vars in 1282 bytes} [Tue Aug 30 05:02:45 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 354 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 30 05:02:45.483682 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-387d4712-a007-4eb6-9787-49a1bb7b6570 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:02:45.594218 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-387d4712-a007-4eb6-9787-49a1bb7b6570 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:02:45.609588 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:02:45.635081 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16547401-67f9-4a86-a390-411279338ba4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:02:45.635704 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16547401-67f9-4a86-a390-411279338ba4 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:45.636071 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16547401-67f9-4a86-a390-411279338ba4 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:45.636312 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16547401-67f9-4a86-a390-411279338ba4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:45.647644 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16547401-67f9-4a86-a390-411279338ba4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:02:45.649083 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16547401-67f9-4a86-a390-411279338ba4 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:02:45.665076 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-16547401-67f9-4a86-a390-411279338ba4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 05:02:45.665076 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:02:45.665076 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:02:45.665076 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:02:45.665076 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:02:45.665076 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:02:45.665076 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:02:45.665076 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:02:45.665076 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:02:45.665076 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:02:45.665076 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:02:45.665076 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:02:45.665076 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 05:02:45.665076 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 05:02:45.665076 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 05:02:45.665076 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 05:02:45.666261 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:02:45.666261 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:02:45.666261 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:02:45.666261 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:02:45.666261 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:02:45.666261 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:02:45.666261 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:02:45.666261 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 05:02:45.666261 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:02:45.666261 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:02:45.666261 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:02:45.666261 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:02:45.666261 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:02:45.666261 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:02:45.666261 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:02:45.666261 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:02:45.666261 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 05:02:45.667277 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 05:02:45.667277 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 05:02:45.667277 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:02:45.667277 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 635/1256] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 05:02:45 2022] GET /identity/v3/domains/Default => generated 87 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:02:45.672572 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f42ef2c0-928d-4213-8d1c-027c764742a8 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:02:45.766068 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:02:45.780128 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f42ef2c0-928d-4213-8d1c-027c764742a8 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:02:45.820486 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbe5d294-baa7-4521-96ae-46009105ef19 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:02:45.821437 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbe5d294-baa7-4521-96ae-46009105ef19 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:45.821840 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbe5d294-baa7-4521-96ae-46009105ef19 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:45.822108 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbe5d294-baa7-4521-96ae-46009105ef19 None admin] PATH_INFO: `/v3/domains` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:45.822896 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbe5d294-baa7-4521-96ae-46009105ef19 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:02:45.823195 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbe5d294-baa7-4521-96ae-46009105ef19 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:02:45.824801 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbe5d294-baa7-4521-96ae-46009105ef19 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:02:45.842665 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 622/1257] 10.222.0.77 () {56 vars in 1109 bytes} [Tue Aug 30 05:02:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:02:45.844294 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16547401-67f9-4a86-a390-411279338ba4 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:02:45.960410 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16547401-67f9-4a86-a390-411279338ba4 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:02:46.000709 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d7a50d9-fdfd-4784-b341-81b0380c66b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:02:46.001332 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d7a50d9-fdfd-4784-b341-81b0380c66b6 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:46.001614 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d7a50d9-fdfd-4784-b341-81b0380c66b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:46.001890 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d7a50d9-fdfd-4784-b341-81b0380c66b6 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:46.009225 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d7a50d9-fdfd-4784-b341-81b0380c66b6 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:02:46.010746 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d7a50d9-fdfd-4784-b341-81b0380c66b6 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:02:46.024303 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-6d7a50d9-fdfd-4784-b341-81b0380c66b6 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 05:02:46.024303 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:02:46.024303 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:02:46.024303 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:02:46.024303 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:02:46.024303 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:02:46.024303 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:02:46.024303 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:02:46.024303 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:02:46.024303 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:02:46.024303 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:02:46.024303 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:02:46.024303 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 05:02:46.024303 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 05:02:46.024303 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 05:02:46.025404 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 05:02:46.025404 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:02:46.025404 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:02:46.025404 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:02:46.025404 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:02:46.025404 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:02:46.025404 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:02:46.025404 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:02:46.025404 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 05:02:46.025404 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:02:46.025404 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:02:46.025404 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:02:46.025404 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:02:46.025404 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:02:46.025404 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:02:46.025404 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:02:46.025404 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:02:46.026549 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 05:02:46.026549 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 05:02:46.026549 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 05:02:46.026549 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 05:02:46.026549 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 05:02:46.026549 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 05:02:46.026549 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 05:02:46.026549 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:02:46.026549 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 636/1258] 10.222.0.77 () {56 vars in 1111 bytes} [Tue Aug 30 05:02:45 2022] GET /identity/v3/projects/service => generated 88 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:02:46.031651 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbe5d294-baa7-4521-96ae-46009105ef19 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:02:46.143142 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbe5d294-baa7-4521-96ae-46009105ef19 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:02:46.181166 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02bf516e-e00d-49f6-8f10-0e635b52f9bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:02:46.181770 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02bf516e-e00d-49f6-8f10-0e635b52f9bd None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:46.182051 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02bf516e-e00d-49f6-8f10-0e635b52f9bd None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:46.182324 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02bf516e-e00d-49f6-8f10-0e635b52f9bd None admin] PATH_INFO: `/v3/projects` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:46.183033 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02bf516e-e00d-49f6-8f10-0e635b52f9bd None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:02:46.183337 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02bf516e-e00d-49f6-8f10-0e635b52f9bd None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:02:46.185127 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02bf516e-e00d-49f6-8f10-0e635b52f9bd None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:02:46.198035 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 623/1259] 10.222.0.77 () {56 vars in 1148 bytes} [Tue Aug 30 05:02:46 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:02:46.204140 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d7a50d9-fdfd-4784-b341-81b0380c66b6 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:02:46.233719 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:02:46.313187 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d7a50d9-fdfd-4784-b341-81b0380c66b6 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:02:46.350051 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f576ad96-d3fb-40cc-8408-87278fdb8707 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:02:46.350671 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f576ad96-d3fb-40cc-8408-87278fdb8707 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:46.350983 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f576ad96-d3fb-40cc-8408-87278fdb8707 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:46.351253 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f576ad96-d3fb-40cc-8408-87278fdb8707 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:46.352262 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f576ad96-d3fb-40cc-8408-87278fdb8707 None admin] RBAC: Adding query filter params (role.id=9d9d5d2c270146488249b7ed943280db, scope.project.id=e9919141e3b947ab8d91b55bba4a7de6, user.id=1972959402704b89925001cd8d90cb0e) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:02:46.352568 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f576ad96-d3fb-40cc-8408-87278fdb8707 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:02:46.354016 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f576ad96-d3fb-40cc-8408-87278fdb8707 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:02:46.360138 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 637/1260] 10.222.0.77 () {56 vars in 1374 bytes} [Tue Aug 30 05:02:46 2022] GET /identity/v3/role_assignments?user.id=1972959402704b89925001cd8d90cb0e&scope.project.id=e9919141e3b947ab8d91b55bba4a7de6&role.id=9d9d5d2c270146488249b7ed943280db => generated 259 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:02:48.254117 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae422e7b-c61b-4ca5-8463-c9b000d70ad3 None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:48.254645 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae422e7b-c61b-4ca5-8463-c9b000d70ad3 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:48.254645 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae422e7b-c61b-4ca5-8463-c9b000d70ad3 None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:48.256056 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 624/1261] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:02:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:02:48.261250 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a63ade-3233-4854-8212-2178c5b5b230 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:48.261683 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a63ade-3233-4854-8212-2178c5b5b230 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:48.261805 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a63ade-3233-4854-8212-2178c5b5b230 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:48.280126 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-36a63ade-3233-4854-8212-2178c5b5b230 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:02:48.281309 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-36a63ade-3233-4854-8212-2178c5b5b230 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:02:48.346784 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 638/1262] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:02:48 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 05:02:48.614085 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae422e7b-c61b-4ca5-8463-c9b000d70ad3 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:02:48.624989 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:02:48.713147 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae422e7b-c61b-4ca5-8463-c9b000d70ad3 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:02:48.745073 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-144ad3e6-9360-4323-b84a-c6f567cee1ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:02:48.745671 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-144ad3e6-9360-4323-b84a-c6f567cee1ef None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:48.745946 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-144ad3e6-9360-4323-b84a-c6f567cee1ef None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:48.746212 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-144ad3e6-9360-4323-b84a-c6f567cee1ef None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:48.756337 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-144ad3e6-9360-4323-b84a-c6f567cee1ef None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 05:02:48.761031 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-144ad3e6-9360-4323-b84a-c6f567cee1ef None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 05:02:48.761031 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:02:48.761031 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:02:48.761031 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:02:48.761031 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:02:48.761031 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:02:48.761031 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:02:48.761031 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:02:48.761031 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:02:48.761031 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:02:48.761031 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:02:48.761031 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:02:48.761031 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 05:02:48.761031 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 05:02:48.761031 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 05:02:48.761031 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 05:02:48.762103 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 05:02:48.762103 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 05:02:48.762103 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 05:02:48.762103 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 05:02:48.762103 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:02:48.762103 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:02:48.762103 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:02:48.762103 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:02:48.762103 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:02:48.762103 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:02:48.762103 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:02:48.762103 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 05:02:48.762103 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:02:48.762103 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:02:48.762103 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:02:48.762103 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:02:48.763118 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:02:48.763118 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:02:48.763118 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:02:48.763118 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:02:48.763118 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 05:02:48.763118 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 05:02:48.763118 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 05:02:48.763118 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 05:02:48.763118 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 05:02:48.763118 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 05:02:48.763118 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 05:02:48.763118 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:02:48.763118 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 625/1263] 10.222.0.77 () {56 vars in 1096 bytes} [Tue Aug 30 05:02:48 2022] GET /identity/v3/roles/admin => generated 83 bytes in 149 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:02:48.767927 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36a63ade-3233-4854-8212-2178c5b5b230 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:02:48.786256 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:02:48.863940 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36a63ade-3233-4854-8212-2178c5b5b230 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:02:48.896688 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74a33086-b883-40dc-b3ff-b3eb81483e8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:02:48.897218 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74a33086-b883-40dc-b3ff-b3eb81483e8c None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:48.897491 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74a33086-b883-40dc-b3ff-b3eb81483e8c None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:48.897752 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74a33086-b883-40dc-b3ff-b3eb81483e8c None admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:48.898394 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74a33086-b883-40dc-b3ff-b3eb81483e8c None admin] RBAC: Adding query filter params (name=admin) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:02:48.898687 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74a33086-b883-40dc-b3ff-b3eb81483e8c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:02:48.900051 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74a33086-b883-40dc-b3ff-b3eb81483e8c None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:02:48.907676 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 639/1264] 10.222.0.77 () {56 vars in 1099 bytes} [Tue Aug 30 05:02:48 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:02:48.913907 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-144ad3e6-9360-4323-b84a-c6f567cee1ef None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:02:48.993908 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:02:49.026640 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-144ad3e6-9360-4323-b84a-c6f567cee1ef None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:02:49.058576 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40587bbd-d520-4297-a919-958657fd69bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:02:49.059217 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40587bbd-d520-4297-a919-958657fd69bf None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:49.059497 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40587bbd-d520-4297-a919-958657fd69bf None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:49.059766 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40587bbd-d520-4297-a919-958657fd69bf None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:49.070390 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40587bbd-d520-4297-a919-958657fd69bf None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:02:49.071845 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40587bbd-d520-4297-a919-958657fd69bf None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:02:49.085435 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-40587bbd-d520-4297-a919-958657fd69bf None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 05:02:49.085435 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:02:49.085435 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:02:49.085435 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:02:49.085435 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:02:49.085435 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:02:49.085435 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:02:49.085435 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:02:49.085435 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:02:49.085435 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:02:49.085435 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:02:49.085435 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:02:49.085435 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 05:02:49.085435 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 05:02:49.085435 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 05:02:49.085435 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 05:02:49.086507 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:02:49.086507 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:02:49.086507 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:02:49.086507 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:02:49.086507 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:02:49.086507 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:02:49.086507 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:02:49.086507 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 05:02:49.086507 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:02:49.086507 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:02:49.086507 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:02:49.086507 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:02:49.086507 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:02:49.086507 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:02:49.086507 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:02:49.086507 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:02:49.086507 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 05:02:49.087441 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 05:02:49.087441 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 05:02:49.087441 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:02:49.087441 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 626/1265] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 05:02:48 2022] GET /identity/v3/domains/Default => generated 87 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:02:49.092396 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74a33086-b883-40dc-b3ff-b3eb81483e8c None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:02:49.170583 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:02:49.189730 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74a33086-b883-40dc-b3ff-b3eb81483e8c None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:02:49.220950 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d48c5b37-c104-4b49-8036-8f000d5317e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:02:49.221525 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d48c5b37-c104-4b49-8036-8f000d5317e0 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:49.221798 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d48c5b37-c104-4b49-8036-8f000d5317e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:49.222062 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d48c5b37-c104-4b49-8036-8f000d5317e0 None admin] PATH_INFO: `/v3/domains` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:49.222681 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d48c5b37-c104-4b49-8036-8f000d5317e0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:02:49.223003 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d48c5b37-c104-4b49-8036-8f000d5317e0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:02:49.224828 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d48c5b37-c104-4b49-8036-8f000d5317e0 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:02:49.236299 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 640/1266] 10.222.0.77 () {56 vars in 1109 bytes} [Tue Aug 30 05:02:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:02:49.242484 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40587bbd-d520-4297-a919-958657fd69bf None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:02:49.343225 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40587bbd-d520-4297-a919-958657fd69bf None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:02:49.362531 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:02:49.376796 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfff77f1-d206-4de0-ad36-97394091cd23 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:02:49.377405 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfff77f1-d206-4de0-ad36-97394091cd23 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:49.377682 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfff77f1-d206-4de0-ad36-97394091cd23 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:49.377951 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfff77f1-d206-4de0-ad36-97394091cd23 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:49.385109 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfff77f1-d206-4de0-ad36-97394091cd23 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:02:49.386538 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfff77f1-d206-4de0-ad36-97394091cd23 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:02:49.397352 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-bfff77f1-d206-4de0-ad36-97394091cd23 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Aug 30 05:02:49.397352 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:02:49.397352 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:02:49.397352 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:02:49.397352 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:02:49.397352 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:02:49.397352 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:02:49.397352 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:02:49.397352 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:02:49.397352 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:02:49.397352 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:02:49.397352 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:02:49.397352 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 05:02:49.397352 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 05:02:49.397352 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 05:02:49.397352 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 05:02:49.398426 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:02:49.398426 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:02:49.398426 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 05:02:49.398426 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 05:02:49.398426 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 05:02:49.398426 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 05:02:49.398426 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:02:49.398426 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:02:49.398426 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:02:49.398426 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:02:49.398426 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:02:49.398426 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 05:02:49.398426 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:02:49.398426 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:02:49.398426 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:02:49.398426 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:02:49.398426 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:02:49.398426 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:02:49.399644 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:02:49.399644 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:02:49.399644 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 05:02:49.399644 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 05:02:49.399644 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 05:02:49.399644 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 05:02:49.399644 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 05:02:49.399644 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 05:02:49.399644 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Aug 30 05:02:49.399644 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:02:49.399644 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 627/1267] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 05:02:49 2022] GET /identity/v3/users/placement => generated 87 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:02:49.404246 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d48c5b37-c104-4b49-8036-8f000d5317e0 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:02:49.503106 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d48c5b37-c104-4b49-8036-8f000d5317e0 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:02:49.535758 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba584178-0490-4771-adc2-9e2320fede11 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:02:49.536927 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba584178-0490-4771-adc2-9e2320fede11 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:49.537199 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba584178-0490-4771-adc2-9e2320fede11 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:49.537463 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba584178-0490-4771-adc2-9e2320fede11 None admin] PATH_INFO: `/v3/users` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:49.538169 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba584178-0490-4771-adc2-9e2320fede11 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:02:49.538463 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba584178-0490-4771-adc2-9e2320fede11 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:02:49.539926 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba584178-0490-4771-adc2-9e2320fede11 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:02:49.548257 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:02:49.557744 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 641/1268] 10.222.0.77 () {56 vars in 1143 bytes} [Tue Aug 30 05:02:49 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:02:49.563587 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfff77f1-d206-4de0-ad36-97394091cd23 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:02:49.660733 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfff77f1-d206-4de0-ad36-97394091cd23 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:02:49.693140 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80e4c1c9-1425-45cc-b215-34c24e3e9947 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:02:49.693740 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80e4c1c9-1425-45cc-b215-34c24e3e9947 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:49.694018 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80e4c1c9-1425-45cc-b215-34c24e3e9947 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:49.694288 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80e4c1c9-1425-45cc-b215-34c24e3e9947 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:49.704416 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80e4c1c9-1425-45cc-b215-34c24e3e9947 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:02:49.705846 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80e4c1c9-1425-45cc-b215-34c24e3e9947 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:02:49.720099 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-80e4c1c9-1425-45cc-b215-34c24e3e9947 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 05:02:49.720099 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:02:49.720099 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:02:49.720099 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:02:49.720099 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:02:49.720099 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:02:49.720099 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:02:49.720099 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:02:49.720099 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:02:49.720099 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:02:49.720099 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:02:49.720099 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:02:49.720099 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 05:02:49.720099 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 05:02:49.720099 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 05:02:49.720099 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 05:02:49.726862 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:02:49.726862 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:02:49.726862 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:02:49.726862 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:02:49.726862 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:02:49.726862 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:02:49.726862 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:02:49.726862 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 05:02:49.726862 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:02:49.726862 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:02:49.726862 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:02:49.726862 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:02:49.726862 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:02:49.726862 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:02:49.726862 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:02:49.726862 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:02:49.726862 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 05:02:49.727982 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 05:02:49.727982 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 05:02:49.727982 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:02:49.727982 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 628/1269] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 05:02:49 2022] GET /identity/v3/domains/Default => generated 87 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:02:49.728325 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba584178-0490-4771-adc2-9e2320fede11 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:02:49.822419 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba584178-0490-4771-adc2-9e2320fede11 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:02:49.855429 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-153048b5-dacc-4372-9d39-f610a6582ee2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:02:49.856008 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-153048b5-dacc-4372-9d39-f610a6582ee2 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:49.856281 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-153048b5-dacc-4372-9d39-f610a6582ee2 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:49.856544 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-153048b5-dacc-4372-9d39-f610a6582ee2 None admin] PATH_INFO: `/v3/domains` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:49.857161 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-153048b5-dacc-4372-9d39-f610a6582ee2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:02:49.857454 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-153048b5-dacc-4372-9d39-f610a6582ee2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:02:49.858782 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-153048b5-dacc-4372-9d39-f610a6582ee2 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:02:49.869417 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 642/1270] 10.222.0.77 () {56 vars in 1109 bytes} [Tue Aug 30 05:02:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:02:49.874963 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80e4c1c9-1425-45cc-b215-34c24e3e9947 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:02:49.905757 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:02:49.980085 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80e4c1c9-1425-45cc-b215-34c24e3e9947 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:02:50.012429 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d134e0a-c85b-4dbb-8f75-3e97582bff84 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:02:50.013039 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d134e0a-c85b-4dbb-8f75-3e97582bff84 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:50.013315 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d134e0a-c85b-4dbb-8f75-3e97582bff84 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:50.013585 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d134e0a-c85b-4dbb-8f75-3e97582bff84 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:50.019931 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d134e0a-c85b-4dbb-8f75-3e97582bff84 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:02:50.021908 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d134e0a-c85b-4dbb-8f75-3e97582bff84 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:02:50.032774 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-4d134e0a-c85b-4dbb-8f75-3e97582bff84 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 05:02:50.032774 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:02:50.032774 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:02:50.032774 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:02:50.032774 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:02:50.032774 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:02:50.032774 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:02:50.032774 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:02:50.032774 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:02:50.032774 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:02:50.032774 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:02:50.032774 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:02:50.032774 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 05:02:50.032774 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 05:02:50.032774 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 05:02:50.033785 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 05:02:50.033785 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:02:50.033785 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:02:50.033785 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:02:50.033785 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:02:50.033785 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:02:50.033785 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:02:50.033785 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:02:50.033785 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 05:02:50.033785 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:02:50.033785 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:02:50.033785 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:02:50.033785 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:02:50.033785 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:02:50.033785 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:02:50.033785 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:02:50.033785 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:02:50.034911 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 05:02:50.034911 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 05:02:50.034911 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 05:02:50.034911 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 05:02:50.034911 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 05:02:50.034911 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 05:02:50.034911 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 05:02:50.034911 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:02:50.034911 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 629/1271] 10.222.0.77 () {56 vars in 1111 bytes} [Tue Aug 30 05:02:49 2022] GET /identity/v3/projects/service => generated 88 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:02:50.039804 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-153048b5-dacc-4372-9d39-f610a6582ee2 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:02:50.096209 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:02:50.136890 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-153048b5-dacc-4372-9d39-f610a6582ee2 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:02:50.168902 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e7c5b66-b3bd-4b84-b992-3b3b8252b0dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:02:50.169475 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7c5b66-b3bd-4b84-b992-3b3b8252b0dc None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:50.169747 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7c5b66-b3bd-4b84-b992-3b3b8252b0dc None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:50.170010 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7c5b66-b3bd-4b84-b992-3b3b8252b0dc None admin] PATH_INFO: `/v3/projects` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:50.170678 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e7c5b66-b3bd-4b84-b992-3b3b8252b0dc None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:02:50.171000 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e7c5b66-b3bd-4b84-b992-3b3b8252b0dc None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:02:50.172402 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e7c5b66-b3bd-4b84-b992-3b3b8252b0dc None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:02:50.183717 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 643/1272] 10.222.0.77 () {56 vars in 1148 bytes} [Tue Aug 30 05:02:50 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:02:50.189733 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d134e0a-c85b-4dbb-8f75-3e97582bff84 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:02:50.280186 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:02:50.289052 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d134e0a-c85b-4dbb-8f75-3e97582bff84 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:02:50.322300 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7dc433c-c5ed-41fa-9db3-051baa8ac931 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:02:50.322956 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7dc433c-c5ed-41fa-9db3-051baa8ac931 None admin] REQUEST_METHOD: `PUT` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:50.323238 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7dc433c-c5ed-41fa-9db3-051baa8ac931 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:50.323508 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7dc433c-c5ed-41fa-9db3-051baa8ac931 None admin] PATH_INFO: `/v3/projects/e9919141e3b947ab8d91b55bba4a7de6/users/1972959402704b89925001cd8d90cb0e/roles/9d9d5d2c270146488249b7ed943280db` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:50.325362 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7dc433c-c5ed-41fa-9db3-051baa8ac931 None admin] RBAC: Authorizing `identity:create_grant(project_id=e9919141e3b947ab8d91b55bba4a7de6, user_id=1972959402704b89925001cd8d90cb0e, role_id=9d9d5d2c270146488249b7ed943280db)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:02:50.326785 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7dc433c-c5ed-41fa-9db3-051baa8ac931 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:02:50.338633 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 630/1273] 10.222.0.77 () {58 vars in 1439 bytes} [Tue Aug 30 05:02:50 2022] PUT /identity/v3/projects/e9919141e3b947ab8d91b55bba4a7de6/users/1972959402704b89925001cd8d90cb0e/roles/9d9d5d2c270146488249b7ed943280db => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:02:52.215761 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62c3df82-43dd-4fa5-a0a0-67c96366bb0a None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:52.216290 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62c3df82-43dd-4fa5-a0a0-67c96366bb0a None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:52.216290 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62c3df82-43dd-4fa5-a0a0-67c96366bb0a None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:52.217617 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 644/1274] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:02:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:02:52.222660 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59324edd-7d51-4c74-ac0c-40c312ce32f5 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:52.223112 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59324edd-7d51-4c74-ac0c-40c312ce32f5 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:52.223245 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59324edd-7d51-4c74-ac0c-40c312ce32f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:52.242263 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-59324edd-7d51-4c74-ac0c-40c312ce32f5 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:02:52.243473 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-59324edd-7d51-4c74-ac0c-40c312ce32f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:02:52.307732 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 631/1275] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:02:52 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 05:02:52.573003 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62c3df82-43dd-4fa5-a0a0-67c96366bb0a None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:02:52.671819 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62c3df82-43dd-4fa5-a0a0-67c96366bb0a None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:02:52.688455 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:02:52.704888 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d9560ae-6555-4bba-ad81-9bb4f575a250 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:02:52.705501 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d9560ae-6555-4bba-ad81-9bb4f575a250 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:52.705784 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d9560ae-6555-4bba-ad81-9bb4f575a250 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:52.706063 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d9560ae-6555-4bba-ad81-9bb4f575a250 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:52.716421 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d9560ae-6555-4bba-ad81-9bb4f575a250 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 05:02:52.721209 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-6d9560ae-6555-4bba-ad81-9bb4f575a250 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 05:02:52.721209 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:02:52.721209 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:02:52.721209 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:02:52.721209 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:02:52.721209 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:02:52.721209 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:02:52.721209 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:02:52.721209 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:02:52.721209 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:02:52.721209 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:02:52.721209 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:02:52.721209 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 05:02:52.721209 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 05:02:52.721209 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 05:02:52.721209 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 05:02:52.722146 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 05:02:52.722146 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 05:02:52.722146 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 05:02:52.722146 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 05:02:52.722146 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:02:52.722146 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:02:52.722146 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:02:52.722146 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:02:52.722146 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:02:52.722146 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:02:52.722146 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:02:52.722146 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 05:02:52.722146 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:02:52.722146 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:02:52.722146 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:02:52.722146 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:02:52.723093 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:02:52.723093 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:02:52.723093 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:02:52.723093 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:02:52.723093 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 05:02:52.723093 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 05:02:52.723093 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 05:02:52.723093 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 05:02:52.723093 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 05:02:52.723093 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 05:02:52.723093 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 05:02:52.723093 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:02:52.723093 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 645/1276] 10.222.0.77 () {56 vars in 1096 bytes} [Tue Aug 30 05:02:52 2022] GET /identity/v3/roles/admin => generated 83 bytes in 150 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:02:52.728972 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59324edd-7d51-4c74-ac0c-40c312ce32f5 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:02:52.807265 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:02:52.825411 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59324edd-7d51-4c74-ac0c-40c312ce32f5 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:02:52.859080 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ec47421-2fa4-4ffd-96d7-23e8394f14a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:02:52.859667 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ec47421-2fa4-4ffd-96d7-23e8394f14a8 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:52.859949 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ec47421-2fa4-4ffd-96d7-23e8394f14a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:52.860222 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ec47421-2fa4-4ffd-96d7-23e8394f14a8 None admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:52.860877 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ec47421-2fa4-4ffd-96d7-23e8394f14a8 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:02:52.861178 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ec47421-2fa4-4ffd-96d7-23e8394f14a8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:02:52.862539 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ec47421-2fa4-4ffd-96d7-23e8394f14a8 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:02:52.870307 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 632/1277] 10.222.0.77 () {56 vars in 1099 bytes} [Tue Aug 30 05:02:52 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:02:52.875760 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d9560ae-6555-4bba-ad81-9bb4f575a250 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:02:52.989037 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d9560ae-6555-4bba-ad81-9bb4f575a250 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:02:53.025832 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8b9c50c-d63a-4552-a7c2-20831ba0f980 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:02:53.026446 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8b9c50c-d63a-4552-a7c2-20831ba0f980 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:53.026721 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8b9c50c-d63a-4552-a7c2-20831ba0f980 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:53.027022 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8b9c50c-d63a-4552-a7c2-20831ba0f980 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:53.036625 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8b9c50c-d63a-4552-a7c2-20831ba0f980 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:02:53.038046 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8b9c50c-d63a-4552-a7c2-20831ba0f980 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:02:53.052015 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-c8b9c50c-d63a-4552-a7c2-20831ba0f980 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 05:02:53.052015 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:02:53.052015 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:02:53.052015 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:02:53.052015 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:02:53.052015 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:02:53.052015 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:02:53.052015 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:02:53.052015 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:02:53.052015 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:02:53.052015 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:02:53.052015 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:02:53.052015 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 05:02:53.052015 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 05:02:53.052015 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 05:02:53.052015 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 05:02:53.053096 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:02:53.053096 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:02:53.053096 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:02:53.053096 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:02:53.053096 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:02:53.053096 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:02:53.053096 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:02:53.053096 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 05:02:53.053096 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:02:53.053096 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:02:53.053096 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:02:53.053096 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:02:53.053096 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:02:53.053096 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:02:53.053096 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:02:53.053096 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:02:53.053096 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 05:02:53.054023 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 05:02:53.054023 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 05:02:53.054023 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:02:53.054023 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 646/1278] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 05:02:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:02:53.059063 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ec47421-2fa4-4ffd-96d7-23e8394f14a8 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:02:53.155622 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ec47421-2fa4-4ffd-96d7-23e8394f14a8 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:02:53.186958 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fd94c67-eaca-4f69-acc7-ff9a97505969 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:02:53.187545 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fd94c67-eaca-4f69-acc7-ff9a97505969 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:53.187821 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fd94c67-eaca-4f69-acc7-ff9a97505969 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:53.188087 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fd94c67-eaca-4f69-acc7-ff9a97505969 None admin] PATH_INFO: `/v3/domains` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:53.188713 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fd94c67-eaca-4f69-acc7-ff9a97505969 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:02:53.189013 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fd94c67-eaca-4f69-acc7-ff9a97505969 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:02:53.190361 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fd94c67-eaca-4f69-acc7-ff9a97505969 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:02:53.202063 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 633/1279] 10.222.0.77 () {56 vars in 1109 bytes} [Tue Aug 30 05:02:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:02:53.207383 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8b9c50c-d63a-4552-a7c2-20831ba0f980 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:02:53.238444 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:02:53.310025 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8b9c50c-d63a-4552-a7c2-20831ba0f980 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:02:53.342733 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dade6b67-265a-4d39-97c7-6fa489a02374 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:02:53.343369 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dade6b67-265a-4d39-97c7-6fa489a02374 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:53.343647 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dade6b67-265a-4d39-97c7-6fa489a02374 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:53.343917 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dade6b67-265a-4d39-97c7-6fa489a02374 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:53.351102 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dade6b67-265a-4d39-97c7-6fa489a02374 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:02:53.352795 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dade6b67-265a-4d39-97c7-6fa489a02374 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:02:53.363552 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-dade6b67-265a-4d39-97c7-6fa489a02374 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Aug 30 05:02:53.363552 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:02:53.363552 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:02:53.363552 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:02:53.363552 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:02:53.363552 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:02:53.363552 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:02:53.363552 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:02:53.363552 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:02:53.363552 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:02:53.363552 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:02:53.363552 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:02:53.363552 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 05:02:53.363552 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 05:02:53.363552 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 05:02:53.363552 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 05:02:53.365128 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:02:53.365128 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:02:53.365128 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 05:02:53.365128 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 05:02:53.365128 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 05:02:53.365128 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 05:02:53.365128 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:02:53.365128 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:02:53.365128 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:02:53.365128 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:02:53.365128 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:02:53.365128 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 05:02:53.365128 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:02:53.365128 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:02:53.365128 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:02:53.365128 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:02:53.365128 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:02:53.365128 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:02:53.366140 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:02:53.366140 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:02:53.366140 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 05:02:53.366140 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 05:02:53.366140 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 05:02:53.366140 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 05:02:53.366140 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 05:02:53.366140 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 05:02:53.366140 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Aug 30 05:02:53.366140 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:02:53.366140 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 647/1280] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 05:02:53 2022] GET /identity/v3/users/placement => generated 87 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:02:53.371007 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fd94c67-eaca-4f69-acc7-ff9a97505969 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:02:53.374771 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:02:53.468295 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fd94c67-eaca-4f69-acc7-ff9a97505969 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:02:53.501186 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95ec0a6e-7ef1-471f-a631-9b905d75ba46 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:02:53.501775 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95ec0a6e-7ef1-471f-a631-9b905d75ba46 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:53.502054 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95ec0a6e-7ef1-471f-a631-9b905d75ba46 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:53.502323 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95ec0a6e-7ef1-471f-a631-9b905d75ba46 None admin] PATH_INFO: `/v3/users` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:53.503087 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95ec0a6e-7ef1-471f-a631-9b905d75ba46 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:02:53.503388 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95ec0a6e-7ef1-471f-a631-9b905d75ba46 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:02:53.505402 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95ec0a6e-7ef1-471f-a631-9b905d75ba46 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:02:53.521600 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 634/1281] 10.222.0.77 () {56 vars in 1143 bytes} [Tue Aug 30 05:02:53 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:02:53.527305 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dade6b67-265a-4d39-97c7-6fa489a02374 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:02:53.614777 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:02:53.624551 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dade6b67-265a-4d39-97c7-6fa489a02374 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:02:53.656636 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f5a587a-f563-440b-9d0c-17f42b90076c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:02:53.657218 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f5a587a-f563-440b-9d0c-17f42b90076c None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:53.657491 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f5a587a-f563-440b-9d0c-17f42b90076c None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:53.657758 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f5a587a-f563-440b-9d0c-17f42b90076c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:53.722106 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f5a587a-f563-440b-9d0c-17f42b90076c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:02:53.724106 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f5a587a-f563-440b-9d0c-17f42b90076c None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:02:53.808823 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:02:53.818773 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 648/1282] 10.222.0.77 () {58 vars in 1282 bytes} [Tue Aug 30 05:02:53 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 292 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 30 05:02:53.824513 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95ec0a6e-7ef1-471f-a631-9b905d75ba46 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:02:53.884458 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:02:53.921518 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95ec0a6e-7ef1-471f-a631-9b905d75ba46 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:02:53.955227 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-692240fb-e53f-47a2-a01f-1c9c058c8c01 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:02:53.955838 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-692240fb-e53f-47a2-a01f-1c9c058c8c01 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:53.956114 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-692240fb-e53f-47a2-a01f-1c9c058c8c01 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:53.956384 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-692240fb-e53f-47a2-a01f-1c9c058c8c01 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:53.966101 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-692240fb-e53f-47a2-a01f-1c9c058c8c01 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:02:53.967553 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-692240fb-e53f-47a2-a01f-1c9c058c8c01 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:02:53.982377 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-692240fb-e53f-47a2-a01f-1c9c058c8c01 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 05:02:53.982377 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:02:53.982377 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:02:53.982377 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:02:53.982377 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:02:53.982377 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:02:53.982377 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:02:53.982377 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:02:53.982377 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:02:53.982377 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:02:53.982377 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:02:53.982377 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:02:53.982377 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 05:02:53.982377 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 05:02:53.982377 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 05:02:53.982377 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 05:02:53.983358 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:02:53.983358 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:02:53.983358 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:02:53.983358 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:02:53.983358 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:02:53.983358 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:02:53.983358 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:02:53.983358 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 05:02:53.983358 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:02:53.983358 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:02:53.983358 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:02:53.983358 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:02:53.983358 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:02:53.983358 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:02:53.983358 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:02:53.983358 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:02:53.983358 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 05:02:53.984375 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 05:02:53.984375 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 05:02:53.984375 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:02:53.984375 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 635/1283] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 05:02:53 2022] GET /identity/v3/domains/Default => generated 87 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:02:53.988981 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f5a587a-f563-440b-9d0c-17f42b90076c None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:02:54.084214 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f5a587a-f563-440b-9d0c-17f42b90076c None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:02:54.117251 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7854f980-3e16-4088-9b85-784141d6d27c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:02:54.117836 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7854f980-3e16-4088-9b85-784141d6d27c None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:54.118112 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7854f980-3e16-4088-9b85-784141d6d27c None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:54.118382 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7854f980-3e16-4088-9b85-784141d6d27c None admin] PATH_INFO: `/v3/domains` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:54.119039 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7854f980-3e16-4088-9b85-784141d6d27c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:02:54.119338 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7854f980-3e16-4088-9b85-784141d6d27c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:02:54.120686 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7854f980-3e16-4088-9b85-784141d6d27c None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:02:54.131459 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 649/1284] 10.222.0.77 () {56 vars in 1109 bytes} [Tue Aug 30 05:02:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:02:54.136948 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-692240fb-e53f-47a2-a01f-1c9c058c8c01 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:02:54.234961 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-692240fb-e53f-47a2-a01f-1c9c058c8c01 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:02:54.247770 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:02:54.267071 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9556e14a-93ad-4066-a7f0-a646412233cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:02:54.267678 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9556e14a-93ad-4066-a7f0-a646412233cb None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:54.267951 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9556e14a-93ad-4066-a7f0-a646412233cb None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:54.268216 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9556e14a-93ad-4066-a7f0-a646412233cb None admin] PATH_INFO: `/v3/projects/service` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:54.275054 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9556e14a-93ad-4066-a7f0-a646412233cb None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:02:54.276477 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9556e14a-93ad-4066-a7f0-a646412233cb None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:02:54.287930 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application [None req-9556e14a-93ad-4066-a7f0-a646412233cb None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 05:02:54.287930 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:02:54.287930 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:02:54.287930 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:02:54.287930 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:02:54.287930 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:02:54.287930 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:02:54.287930 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:02:54.287930 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:02:54.287930 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:02:54.287930 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:02:54.287930 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:02:54.287930 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 05:02:54.287930 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 05:02:54.287930 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 05:02:54.289123 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 05:02:54.289123 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:02:54.289123 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:02:54.289123 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:02:54.289123 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:02:54.289123 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:02:54.289123 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:02:54.289123 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:02:54.289123 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application with Lock( Aug 30 05:02:54.289123 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:02:54.289123 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:02:54.289123 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:02:54.289123 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:02:54.289123 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:02:54.289123 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:02:54.289123 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:02:54.289123 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:02:54.290462 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 05:02:54.290462 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 05:02:54.290462 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 05:02:54.290462 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 05:02:54.290462 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 05:02:54.290462 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 05:02:54.290462 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 05:02:54.290462 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: ERROR keystone.server.flask.application Aug 30 05:02:54.290462 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 636/1285] 10.222.0.77 () {56 vars in 1111 bytes} [Tue Aug 30 05:02:54 2022] GET /identity/v3/projects/service => generated 88 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:02:54.295021 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7854f980-3e16-4088-9b85-784141d6d27c None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:02:54.342842 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:02:54.392734 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7854f980-3e16-4088-9b85-784141d6d27c None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:02:54.426131 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83dc73f0-20f4-4b0b-bd96-888380064143 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:02:54.426720 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83dc73f0-20f4-4b0b-bd96-888380064143 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:54.427028 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83dc73f0-20f4-4b0b-bd96-888380064143 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:54.427298 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83dc73f0-20f4-4b0b-bd96-888380064143 None admin] PATH_INFO: `/v3/projects` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:54.427972 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83dc73f0-20f4-4b0b-bd96-888380064143 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:02:54.428272 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83dc73f0-20f4-4b0b-bd96-888380064143 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:02:54.429686 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83dc73f0-20f4-4b0b-bd96-888380064143 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:02:54.441795 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 650/1286] 10.222.0.77 () {56 vars in 1148 bytes} [Tue Aug 30 05:02:54 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 05:02:54.448332 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9556e14a-93ad-4066-a7f0-a646412233cb None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:02:54.544728 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9556e14a-93ad-4066-a7f0-a646412233cb None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:02:54.576531 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cd56580-edba-456c-a43f-5232e36bfc91 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:02:54.577109 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cd56580-edba-456c-a43f-5232e36bfc91 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:54.577382 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cd56580-edba-456c-a43f-5232e36bfc91 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:54.577647 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cd56580-edba-456c-a43f-5232e36bfc91 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:54.578351 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cd56580-edba-456c-a43f-5232e36bfc91 None admin] RBAC: Adding query filter params (role.id=9d9d5d2c270146488249b7ed943280db, scope.project.id=e9919141e3b947ab8d91b55bba4a7de6, user.id=1972959402704b89925001cd8d90cb0e) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:02:54.578650 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cd56580-edba-456c-a43f-5232e36bfc91 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:02:54.580079 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cd56580-edba-456c-a43f-5232e36bfc91 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:02:54.586216 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 637/1287] 10.222.0.77 () {56 vars in 1374 bytes} [Tue Aug 30 05:02:54 2022] GET /identity/v3/role_assignments?user.id=1972959402704b89925001cd8d90cb0e&scope.project.id=e9919141e3b947ab8d91b55bba4a7de6&role.id=9d9d5d2c270146488249b7ed943280db => generated 608 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:02:56.467128 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-669ff52c-62d7-428e-9650-df20d08cfd41 None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:56.467526 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-669ff52c-62d7-428e-9650-df20d08cfd41 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:56.467669 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-669ff52c-62d7-428e-9650-df20d08cfd41 None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:56.469001 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 651/1288] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:02:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:02:56.473999 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2edf7d40-f388-4a78-b6d3-2515393c6626 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:56.474395 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2edf7d40-f388-4a78-b6d3-2515393c6626 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:56.474554 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2edf7d40-f388-4a78-b6d3-2515393c6626 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:56.493142 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-2edf7d40-f388-4a78-b6d3-2515393c6626 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:02:56.494352 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-2edf7d40-f388-4a78-b6d3-2515393c6626 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:02:56.523331 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:02:56.560125 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 638/1289] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:02:56 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 05:02:56.824060 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-669ff52c-62d7-428e-9650-df20d08cfd41 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:02:56.924505 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-669ff52c-62d7-428e-9650-df20d08cfd41 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:02:56.938073 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:02:56.966336 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a66059a-9265-4ff2-ac7d-061e0b73e28a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:02:56.967278 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a66059a-9265-4ff2-ac7d-061e0b73e28a None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:56.967550 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a66059a-9265-4ff2-ac7d-061e0b73e28a None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:56.967936 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a66059a-9265-4ff2-ac7d-061e0b73e28a None admin] PATH_INFO: `/v3/services/placement` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:56.968727 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a66059a-9265-4ff2-ac7d-061e0b73e28a None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:02:56.970473 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a66059a-9265-4ff2-ac7d-061e0b73e28a None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:02:56.982242 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-6a66059a-9265-4ff2-ac7d-061e0b73e28a None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Aug 30 05:02:56.982242 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:02:56.982242 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 30 05:02:56.982242 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 30 05:02:56.982242 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 30 05:02:56.982242 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 30 05:02:56.982242 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 30 05:02:56.982242 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 05:02:56.982242 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Aug 30 05:02:56.982242 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:02:56.982242 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 05:02:56.982242 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:02:56.982242 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:02:56.982242 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:02:56.982242 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:02:56.982242 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:02:56.982242 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:02:56.983435 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:02:56.983435 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:02:56.983435 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:02:56.983435 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:02:56.983435 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:02:56.983435 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:02:56.983435 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 30 05:02:56.983435 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 30 05:02:56.983435 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 30 05:02:56.983435 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 30 05:02:56.983435 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:02:56.983435 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:02:56.983435 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:02:56.983435 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:02:56.983435 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:02:56.983435 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:02:56.984313 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:02:56.984313 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 05:02:56.984313 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:02:56.984313 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:02:56.984313 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:02:56.984313 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:02:56.984313 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:02:56.984313 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:02:56.984313 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:02:56.984313 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:02:56.984313 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 30 05:02:56.984313 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 05:02:56.984313 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Aug 30 05:02:56.984313 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:02:56.984313 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 652/1290] 10.222.0.77 () {56 vars in 1117 bytes} [Tue Aug 30 05:02:56 2022] GET /identity/v3/services/placement => generated 90 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:02:56.990512 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2edf7d40-f388-4a78-b6d3-2515393c6626 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:02:57.085648 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2edf7d40-f388-4a78-b6d3-2515393c6626 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:02:57.118517 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21e3cfe8-c110-435a-8c74-b5c944427fc3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:02:57.119128 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21e3cfe8-c110-435a-8c74-b5c944427fc3 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:57.119403 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21e3cfe8-c110-435a-8c74-b5c944427fc3 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:57.119668 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21e3cfe8-c110-435a-8c74-b5c944427fc3 None admin] PATH_INFO: `/v3/services` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:57.120278 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21e3cfe8-c110-435a-8c74-b5c944427fc3 None admin] RBAC: Adding query filter params (name=placement) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:02:57.120572 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21e3cfe8-c110-435a-8c74-b5c944427fc3 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:02:57.121924 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21e3cfe8-c110-435a-8c74-b5c944427fc3 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:02:57.126503 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 639/1291] 10.222.0.77 () {56 vars in 1116 bytes} [Tue Aug 30 05:02:56 2022] GET /identity/v3/services?name=placement => generated 126 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:02:57.132819 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a66059a-9265-4ff2-ac7d-061e0b73e28a None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:02:57.228665 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a66059a-9265-4ff2-ac7d-061e0b73e28a None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:02:57.262023 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ab1c8c2-8230-4df0-9725-ee678e46664c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:02:57.262614 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab1c8c2-8230-4df0-9725-ee678e46664c None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:57.262919 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab1c8c2-8230-4df0-9725-ee678e46664c None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:57.263191 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab1c8c2-8230-4df0-9725-ee678e46664c None admin] PATH_INFO: `/v3/services` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:57.263806 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ab1c8c2-8230-4df0-9725-ee678e46664c None admin] RBAC: Adding query filter params (type=placement) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:02:57.264104 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ab1c8c2-8230-4df0-9725-ee678e46664c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:02:57.265451 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ab1c8c2-8230-4df0-9725-ee678e46664c None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:02:57.269722 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 653/1292] 10.222.0.77 () {56 vars in 1116 bytes} [Tue Aug 30 05:02:57 2022] GET /identity/v3/services?type=placement => generated 126 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:02:59.151121 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-180cc760-60b9-4d5a-ba67-69363dd5b8cd None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:59.151649 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-180cc760-60b9-4d5a-ba67-69363dd5b8cd None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:59.151649 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-180cc760-60b9-4d5a-ba67-69363dd5b8cd None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:59.153012 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 640/1293] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:02:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:02:59.158030 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3987039c-9b80-4982-af42-bc97f8771943 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:59.158404 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3987039c-9b80-4982-af42-bc97f8771943 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:59.158570 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3987039c-9b80-4982-af42-bc97f8771943 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:59.176529 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-3987039c-9b80-4982-af42-bc97f8771943 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:02:59.177699 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-3987039c-9b80-4982-af42-bc97f8771943 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:02:59.206292 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:02:59.242707 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 654/1294] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:02:59 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 05:02:59.516152 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-180cc760-60b9-4d5a-ba67-69363dd5b8cd None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:02:59.546333 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:02:59.616134 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-180cc760-60b9-4d5a-ba67-69363dd5b8cd None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:02:59.648829 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-993a69f0-4984-4f7d-b97d-c03953062a1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:02:59.649416 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-993a69f0-4984-4f7d-b97d-c03953062a1b None admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:02:59.649695 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-993a69f0-4984-4f7d-b97d-c03953062a1b None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:02:59.649965 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-993a69f0-4984-4f7d-b97d-c03953062a1b None admin] PATH_INFO: `/v3/services` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:02:59.650600 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-993a69f0-4984-4f7d-b97d-c03953062a1b None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:02:59.651987 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-993a69f0-4984-4f7d-b97d-c03953062a1b None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:02:59.665935 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 641/1295] 10.222.0.77 () {60 vars in 1141 bytes} [Tue Aug 30 05:02:59 2022] POST /identity/v3/services => generated 246 bytes in 150 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:03:01.566304 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a496dfa2-129d-4ae2-9c3f-06627d9cd3a3 None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:03:01.566794 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a496dfa2-129d-4ae2-9c3f-06627d9cd3a3 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:03:01.566945 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a496dfa2-129d-4ae2-9c3f-06627d9cd3a3 None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:03:01.568304 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 655/1296] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:03:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:03:01.573429 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f4b3c90-ff61-469c-8c84-881018feda11 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:03:01.573844 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f4b3c90-ff61-469c-8c84-881018feda11 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:03:01.573980 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f4b3c90-ff61-469c-8c84-881018feda11 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:03:01.592149 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-2f4b3c90-ff61-469c-8c84-881018feda11 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:03:01.593322 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-2f4b3c90-ff61-469c-8c84-881018feda11 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:03:01.663776 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 642/1297] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:03:01 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 05:03:01.928510 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a496dfa2-129d-4ae2-9c3f-06627d9cd3a3 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:03:02.027105 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a496dfa2-129d-4ae2-9c3f-06627d9cd3a3 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:03:02.059065 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aebb08e2-96bb-495a-998e-93c98b874172 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:03:02.059661 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aebb08e2-96bb-495a-998e-93c98b874172 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:03:02.059934 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aebb08e2-96bb-495a-998e-93c98b874172 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:03:02.060199 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aebb08e2-96bb-495a-998e-93c98b874172 None admin] PATH_INFO: `/v3/services/placement` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:03:02.060762 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aebb08e2-96bb-495a-998e-93c98b874172 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:03:02.062106 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aebb08e2-96bb-495a-998e-93c98b874172 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:03:02.071137 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-aebb08e2-96bb-495a-998e-93c98b874172 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Aug 30 05:03:02.071137 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:03:02.071137 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 30 05:03:02.071137 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 30 05:03:02.071137 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 30 05:03:02.071137 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 30 05:03:02.071137 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 30 05:03:02.071137 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 05:03:02.071137 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Aug 30 05:03:02.071137 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:03:02.071137 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 05:03:02.071137 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:03:02.071137 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:03:02.071137 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:03:02.071137 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:03:02.071137 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:03:02.071137 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:03:02.072293 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:03:02.072293 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:03:02.072293 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:03:02.072293 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:03:02.072293 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:03:02.072293 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:03:02.072293 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 30 05:03:02.072293 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 30 05:03:02.072293 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 30 05:03:02.072293 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 30 05:03:02.072293 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:03:02.072293 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:03:02.072293 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:03:02.072293 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:03:02.072293 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:03:02.072293 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:03:02.073233 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:03:02.073233 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 05:03:02.073233 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:03:02.073233 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:03:02.073233 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:03:02.073233 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:03:02.073233 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:03:02.073233 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:03:02.073233 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:03:02.073233 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:03:02.073233 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 30 05:03:02.073233 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 05:03:02.073233 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Aug 30 05:03:02.073233 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:03:02.073233 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 656/1298] 10.222.0.77 () {56 vars in 1117 bytes} [Tue Aug 30 05:03:01 2022] GET /identity/v3/services/placement => generated 90 bytes in 144 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:03:02.078054 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f4b3c90-ff61-469c-8c84-881018feda11 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:03:02.096322 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:03:02.174033 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f4b3c90-ff61-469c-8c84-881018feda11 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:03:02.206422 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c600f35f-c8d0-4c2a-8807-d6393d5f1fd9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:03:02.206981 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c600f35f-c8d0-4c2a-8807-d6393d5f1fd9 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:03:02.207261 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c600f35f-c8d0-4c2a-8807-d6393d5f1fd9 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:03:02.207531 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c600f35f-c8d0-4c2a-8807-d6393d5f1fd9 None admin] PATH_INFO: `/v3/services` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:03:02.208691 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c600f35f-c8d0-4c2a-8807-d6393d5f1fd9 None admin] RBAC: Adding query filter params (name=placement) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:03:02.208987 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c600f35f-c8d0-4c2a-8807-d6393d5f1fd9 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:03:02.210336 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c600f35f-c8d0-4c2a-8807-d6393d5f1fd9 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:03:02.215276 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 643/1299] 10.222.0.77 () {56 vars in 1116 bytes} [Tue Aug 30 05:03:02 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:03:02.221009 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aebb08e2-96bb-495a-998e-93c98b874172 None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:03:02.251581 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:03:02.318694 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aebb08e2-96bb-495a-998e-93c98b874172 None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:03:02.350621 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd67a5a9-4eb9-4ab3-ab9a-7326584da848 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:03:02.351247 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd67a5a9-4eb9-4ab3-ab9a-7326584da848 None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:03:02.351793 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd67a5a9-4eb9-4ab3-ab9a-7326584da848 None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:03:02.352060 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd67a5a9-4eb9-4ab3-ab9a-7326584da848 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:03:02.352739 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd67a5a9-4eb9-4ab3-ab9a-7326584da848 None admin] RBAC: Adding query filter params (interface=public, service_id=b8bd98cefb334f79b9f6d07e0d87149d, region_id=RegionOne) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:03:02.353040 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd67a5a9-4eb9-4ab3-ab9a-7326584da848 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:03:02.354382 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd67a5a9-4eb9-4ab3-ab9a-7326584da848 None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:03:02.359241 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 657/1300] 10.222.0.77 () {56 vars in 1251 bytes} [Tue Aug 30 05:03:02 2022] GET /identity/v3/endpoints?service_id=b8bd98cefb334f79b9f6d07e0d87149d&interface=public®ion_id=RegionOne => generated 194 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:03:02.365175 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c600f35f-c8d0-4c2a-8807-d6393d5f1fd9 None admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:03:02.443623 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:03:02.461777 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c600f35f-c8d0-4c2a-8807-d6393d5f1fd9 None admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:03:02.495265 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-553085a1-86b1-4a86-b21e-f99190286c78 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:03:02.495857 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-553085a1-86b1-4a86-b21e-f99190286c78 None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:03:02.496132 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-553085a1-86b1-4a86-b21e-f99190286c78 None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:03:02.496399 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-553085a1-86b1-4a86-b21e-f99190286c78 None admin] PATH_INFO: `/v3/services` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:03:02.496989 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-553085a1-86b1-4a86-b21e-f99190286c78 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:03:02.498339 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-553085a1-86b1-4a86-b21e-f99190286c78 None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:03:02.505548 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 644/1301] 10.222.0.77 () {56 vars in 1087 bytes} [Tue Aug 30 05:03:02 2022] GET /identity/v3/services => generated 2191 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:03:04.374058 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-242bd7a3-9194-4360-a362-363c32d079a9 None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:03:04.374580 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-242bd7a3-9194-4360-a362-363c32d079a9 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:03:04.374580 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-242bd7a3-9194-4360-a362-363c32d079a9 None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:03:04.375939 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 658/1302] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:03:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:03:04.381230 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f82084a-f105-4673-94c5-8820de02b1e4 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:03:04.381642 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f82084a-f105-4673-94c5-8820de02b1e4 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:03:04.381768 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f82084a-f105-4673-94c5-8820de02b1e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:03:04.399348 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-7f82084a-f105-4673-94c5-8820de02b1e4 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:03:04.400520 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-7f82084a-f105-4673-94c5-8820de02b1e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:03:04.465013 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 645/1303] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:03:04 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 05:03:04.730741 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-242bd7a3-9194-4360-a362-363c32d079a9 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:03:04.822720 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:03:04.829571 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-242bd7a3-9194-4360-a362-363c32d079a9 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:03:04.862367 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fc14eda-1f73-4810-b00f-9c94c977a2de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:03:04.862997 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fc14eda-1f73-4810-b00f-9c94c977a2de None admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:03:04.863270 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fc14eda-1f73-4810-b00f-9c94c977a2de None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:03:04.863535 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fc14eda-1f73-4810-b00f-9c94c977a2de None admin] PATH_INFO: `/v3/services/placement` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:03:04.864121 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fc14eda-1f73-4810-b00f-9c94c977a2de None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:03:04.865465 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fc14eda-1f73-4810-b00f-9c94c977a2de None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:03:04.873867 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application [None req-8fc14eda-1f73-4810-b00f-9c94c977a2de None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Aug 30 05:03:04.873867 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:03:04.873867 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 30 05:03:04.873867 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 30 05:03:04.873867 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 30 05:03:04.873867 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 30 05:03:04.873867 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 30 05:03:04.873867 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 05:03:04.873867 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Aug 30 05:03:04.873867 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:03:04.873867 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 05:03:04.873867 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:03:04.873867 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 05:03:04.873867 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 05:03:04.873867 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 05:03:04.873867 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 05:03:04.873867 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 05:03:04.875065 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 05:03:04.875065 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 05:03:04.875065 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 05:03:04.875065 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 05:03:04.875065 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 05:03:04.875065 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 05:03:04.875065 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 30 05:03:04.875065 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 30 05:03:04.875065 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 30 05:03:04.875065 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 30 05:03:04.875065 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 05:03:04.875065 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 05:03:04.875065 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 05:03:04.875065 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 05:03:04.875065 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 05:03:04.875065 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 05:03:04.876250 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 05:03:04.876250 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application with Lock( Aug 30 05:03:04.876250 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 05:03:04.876250 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self._enter() Aug 30 05:03:04.876250 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 05:03:04.876250 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 05:03:04.876250 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 05:03:04.876250 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application return self.creator() Aug 30 05:03:04.876250 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 05:03:04.876250 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application created_value = creator( Aug 30 05:03:04.876250 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 30 05:03:04.876250 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 05:03:04.876250 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Aug 30 05:03:04.876250 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: ERROR keystone.server.flask.application Aug 30 05:03:04.876250 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 659/1304] 10.222.0.77 () {56 vars in 1117 bytes} [Tue Aug 30 05:03:04 2022] GET /identity/v3/services/placement => generated 90 bytes in 145 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 05:03:04.881276 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f82084a-f105-4673-94c5-8820de02b1e4 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:03:04.965877 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:03:04.992814 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f82084a-f105-4673-94c5-8820de02b1e4 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:03:05.027546 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-267c2247-43c8-49a9-b9e7-efa9b1d84f5d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:03:05.028140 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-267c2247-43c8-49a9-b9e7-efa9b1d84f5d None admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:03:05.028418 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-267c2247-43c8-49a9-b9e7-efa9b1d84f5d None admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:03:05.028689 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-267c2247-43c8-49a9-b9e7-efa9b1d84f5d None admin] PATH_INFO: `/v3/services` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:03:05.029305 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-267c2247-43c8-49a9-b9e7-efa9b1d84f5d None admin] RBAC: Adding query filter params (name=placement) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:03:05.029603 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-267c2247-43c8-49a9-b9e7-efa9b1d84f5d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:03:05.030995 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-267c2247-43c8-49a9-b9e7-efa9b1d84f5d None admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:03:05.035982 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 646/1305] 10.222.0.77 () {56 vars in 1116 bytes} [Tue Aug 30 05:03:04 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:03:05.041717 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fc14eda-1f73-4810-b00f-9c94c977a2de None admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:03:05.136906 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fc14eda-1f73-4810-b00f-9c94c977a2de None admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:03:05.169073 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c698b84c-1530-4829-a70b-1489c420245d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:03:05.169642 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c698b84c-1530-4829-a70b-1489c420245d None admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:03:05.169916 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c698b84c-1530-4829-a70b-1489c420245d None admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:03:05.170182 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c698b84c-1530-4829-a70b-1489c420245d None admin] PATH_INFO: `/v3/endpoints` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:03:05.170844 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c698b84c-1530-4829-a70b-1489c420245d None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:03:05.172180 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c698b84c-1530-4829-a70b-1489c420245d None admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:03:05.190669 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 660/1306] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:03:05 2022] POST /identity/v3/endpoints => generated 331 bytes in 149 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:03:15.033247 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=30 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:03:15.186876 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=91,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:04:07.780689 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84af9ff0-4e98-48e6-9f9a-0c434f7da80c None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:04:07.781226 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84af9ff0-4e98-48e6-9f9a-0c434f7da80c None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:04:07.781226 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84af9ff0-4e98-48e6-9f9a-0c434f7da80c None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:04:07.788259 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d82a5f8-66d1-4020-b9cb-83d52ba11559 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:04:07.788731 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d82a5f8-66d1-4020-b9cb-83d52ba11559 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:04:07.788821 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d82a5f8-66d1-4020-b9cb-83d52ba11559 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:04:07.793550 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 647/1307] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:04:07 2022] GET /identity => generated 270 bytes in 14 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:04:07.877124 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-1d82a5f8-66d1-4020-b9cb-83d52ba11559 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:04:07.878891 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-1d82a5f8-66d1-4020-b9cb-83d52ba11559 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:04:08.074883 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 661/1308] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:04:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 281 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 05:04:08.397441 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca713d00-2974-4683-aca5-188dab41229b None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:04:08.397821 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca713d00-2974-4683-aca5-188dab41229b None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:04:08.398001 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca713d00-2974-4683-aca5-188dab41229b None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:04:08.399722 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 648/1309] 10.222.0.77 () {54 vars in 965 bytes} [Tue Aug 30 05:04:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:04:08.408657 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0611f422-93d5-4266-ada8-1cb0c9359c7d None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:04:08.408938 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0611f422-93d5-4266-ada8-1cb0c9359c7d None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:04:08.409264 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0611f422-93d5-4266-ada8-1cb0c9359c7d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:04:08.495347 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-0611f422-93d5-4266-ada8-1cb0c9359c7d None None] MFA Rules not processed for user `c33c45f4770d4151858e1d897d09af69`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:04:08.496554 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-0611f422-93d5-4266-ada8-1cb0c9359c7d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:04:08.517512 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:04:08.574912 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 662/1310] 10.222.0.77 () {58 vars in 1062 bytes} [Tue Aug 30 05:04:08 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 05:04:08.583647 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca713d00-2974-4683-aca5-188dab41229b None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:04:08.585485 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-ca713d00-2974-4683-aca5-188dab41229b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:04:08.697077 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca713d00-2974-4683-aca5-188dab41229b None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:04:08.780662 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-136fdb90-833d-4e15-a97a-d7232cf10c81 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:04:08.781280 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-136fdb90-833d-4e15-a97a-d7232cf10c81 service neutron] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:04:08.781810 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-136fdb90-833d-4e15-a97a-d7232cf10c81 service neutron] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:04:08.781810 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-136fdb90-833d-4e15-a97a-d7232cf10c81 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:04:08.783479 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-136fdb90-833d-4e15-a97a-d7232cf10c81 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:04:08.845259 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:04:08.906696 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-136fdb90-833d-4e15-a97a-d7232cf10c81 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:04:08.908188 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-136fdb90-833d-4e15-a97a-d7232cf10c81 service neutron] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:04:08.931216 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 649/1311] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:04:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 348 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:04:11.114291 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deab0feb-2a99-48ba-b935-e223709fe527 None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:04:11.114861 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deab0feb-2a99-48ba-b935-e223709fe527 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:04:11.114861 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deab0feb-2a99-48ba-b935-e223709fe527 None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:04:11.120446 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 663/1312] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:04:11 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:04:11.122167 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd51d8d6-0a2f-4bed-9de6-59d9e603594e None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:04:11.122644 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd51d8d6-0a2f-4bed-9de6-59d9e603594e None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:04:11.122865 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd51d8d6-0a2f-4bed-9de6-59d9e603594e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:04:11.210449 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-bd51d8d6-0a2f-4bed-9de6-59d9e603594e None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:04:11.211893 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-bd51d8d6-0a2f-4bed-9de6-59d9e603594e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:04:11.330857 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 650/1313] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:04:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 210 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:04:11.611045 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deab0feb-2a99-48ba-b935-e223709fe527 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:04:11.613098 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-deab0feb-2a99-48ba-b935-e223709fe527 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:04:11.719336 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deab0feb-2a99-48ba-b935-e223709fe527 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:04:11.790863 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:04:11.793794 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f8f076a-98cb-4f09-855f-00b614d072e4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:04:11.794620 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f8f076a-98cb-4f09-855f-00b614d072e4 service neutron] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:04:11.795026 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f8f076a-98cb-4f09-855f-00b614d072e4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:04:11.795498 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f8f076a-98cb-4f09-855f-00b614d072e4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:04:11.797342 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-2f8f076a-98cb-4f09-855f-00b614d072e4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:04:11.905456 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f8f076a-98cb-4f09-855f-00b614d072e4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:04:11.906930 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f8f076a-98cb-4f09-855f-00b614d072e4 service neutron] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:04:11.927630 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 664/1314] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:04:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 317 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:04:14.154889 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ac9e28-2b12-43b5-a3f5-16d71596bc75 None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:04:14.155408 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ac9e28-2b12-43b5-a3f5-16d71596bc75 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:04:14.155530 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ac9e28-2b12-43b5-a3f5-16d71596bc75 None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:04:14.156775 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 651/1315] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:04:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 05:04:14.162607 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-352696b4-50d1-40e3-b3bc-db3d63e0b5fa None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:04:14.163041 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-352696b4-50d1-40e3-b3bc-db3d63e0b5fa None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:04:14.163195 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-352696b4-50d1-40e3-b3bc-db3d63e0b5fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:04:14.227931 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-352696b4-50d1-40e3-b3bc-db3d63e0b5fa None None] MFA Rules not processed for user `bb6b84c639b14608813a9a5e0dac6840`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:04:14.229125 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-352696b4-50d1-40e3-b3bc-db3d63e0b5fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:04:14.280355 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:04:14.324350 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 665/1316] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:04:14 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:04:14.623600 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5feb8431-1af9-4571-b525-54ec2b2ef00d None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:04:14.624040 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5feb8431-1af9-4571-b525-54ec2b2ef00d None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:04:14.624324 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5feb8431-1af9-4571-b525-54ec2b2ef00d None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:04:14.625670 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 652/1317] 10.222.0.77 () {54 vars in 965 bytes} [Tue Aug 30 05:04:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:04:14.632000 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc4db08f-1782-4ab1-8481-8041d598038d None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:04:14.632286 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc4db08f-1782-4ab1-8481-8041d598038d None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:04:14.632556 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc4db08f-1782-4ab1-8481-8041d598038d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:04:14.706562 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-dc4db08f-1782-4ab1-8481-8041d598038d None None] MFA Rules not processed for user `c33c45f4770d4151858e1d897d09af69`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:04:14.707786 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-dc4db08f-1782-4ab1-8481-8041d598038d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:04:14.781860 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 666/1318] 10.222.0.77 () {58 vars in 1062 bytes} [Tue Aug 30 05:04:14 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:04:14.790568 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5feb8431-1af9-4571-b525-54ec2b2ef00d None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:04:14.792110 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-5feb8431-1af9-4571-b525-54ec2b2ef00d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:04:14.801566 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:04:14.894040 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5feb8431-1af9-4571-b525-54ec2b2ef00d None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:04:14.964482 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e319e65-e2a5-44df-9e16-1985bad15d4c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:04:14.965327 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e319e65-e2a5-44df-9e16-1985bad15d4c service neutron] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:04:14.965605 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e319e65-e2a5-44df-9e16-1985bad15d4c service neutron] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:04:14.965869 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e319e65-e2a5-44df-9e16-1985bad15d4c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:04:14.967449 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-3e319e65-e2a5-44df-9e16-1985bad15d4c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:04:15.027889 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:04:15.073690 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e319e65-e2a5-44df-9e16-1985bad15d4c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:04:15.075139 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e319e65-e2a5-44df-9e16-1985bad15d4c service neutron] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:04:15.092328 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 653/1319] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:04:14 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 302 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:04:17.885216 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63945348-d7e4-463f-933a-a9a981bfefb6 None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:04:17.885843 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63945348-d7e4-463f-933a-a9a981bfefb6 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:04:17.885843 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63945348-d7e4-463f-933a-a9a981bfefb6 None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:04:17.887118 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 667/1320] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:04:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:04:17.892262 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de64d3b3-e4db-4b73-b9a6-a99613eac8a2 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:04:17.892684 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de64d3b3-e4db-4b73-b9a6-a99613eac8a2 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:04:17.892797 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de64d3b3-e4db-4b73-b9a6-a99613eac8a2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:04:17.960123 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-de64d3b3-e4db-4b73-b9a6-a99613eac8a2 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:04:17.961320 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-de64d3b3-e4db-4b73-b9a6-a99613eac8a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:04:18.055505 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 654/1321] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:04:17 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:04:18.340089 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63945348-d7e4-463f-933a-a9a981bfefb6 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:04:18.341625 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-63945348-d7e4-463f-933a-a9a981bfefb6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:04:18.391403 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:04:18.445866 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63945348-d7e4-463f-933a-a9a981bfefb6 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:04:18.518646 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dea6484-1548-4708-b088-54479cff23ef service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:04:18.519267 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dea6484-1548-4708-b088-54479cff23ef service neutron] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:04:18.519547 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dea6484-1548-4708-b088-54479cff23ef service neutron] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:04:18.519818 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dea6484-1548-4708-b088-54479cff23ef service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:04:18.521399 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-3dea6484-1548-4708-b088-54479cff23ef service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:04:18.615458 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:04:18.628851 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dea6484-1548-4708-b088-54479cff23ef service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:04:18.630272 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dea6484-1548-4708-b088-54479cff23ef service neutron] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:04:18.647662 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 668/1322] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:04:18 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 308 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:04:21.173595 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fbd4560-81f1-4024-9c5f-015fa82203e8 None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:04:21.174164 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fbd4560-81f1-4024-9c5f-015fa82203e8 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:04:21.174164 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fbd4560-81f1-4024-9c5f-015fa82203e8 None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:04:21.176046 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 655/1323] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:04:21 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:04:21.181072 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9d25656-9e8c-4307-b1b0-25087f8917e3 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:04:21.181496 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9d25656-9e8c-4307-b1b0-25087f8917e3 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:04:21.181622 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9d25656-9e8c-4307-b1b0-25087f8917e3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:04:21.249201 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-f9d25656-9e8c-4307-b1b0-25087f8917e3 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:04:21.250407 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-f9d25656-9e8c-4307-b1b0-25087f8917e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:04:21.344736 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 669/1324] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:04:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:04:21.625096 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fbd4560-81f1-4024-9c5f-015fa82203e8 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:04:21.626691 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-8fbd4560-81f1-4024-9c5f-015fa82203e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:04:21.645899 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:04:21.735218 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fbd4560-81f1-4024-9c5f-015fa82203e8 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:04:21.810432 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f075d54-6bf2-47a4-bc0a-32e7c007b6c9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:04:21.811081 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f075d54-6bf2-47a4-bc0a-32e7c007b6c9 service neutron] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:04:21.811356 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f075d54-6bf2-47a4-bc0a-32e7c007b6c9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:04:21.811622 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f075d54-6bf2-47a4-bc0a-32e7c007b6c9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:04:21.813188 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-2f075d54-6bf2-47a4-bc0a-32e7c007b6c9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:04:21.883037 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:04:21.921681 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f075d54-6bf2-47a4-bc0a-32e7c007b6c9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:04:21.922984 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f075d54-6bf2-47a4-bc0a-32e7c007b6c9 service neutron] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:04:21.941254 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 656/1325] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:04:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 317 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:04:23.873231 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-321253a9-45a9-4442-94a0-e6ff8cabc8c3 None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:04:23.873759 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-321253a9-45a9-4442-94a0-e6ff8cabc8c3 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:04:23.873759 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-321253a9-45a9-4442-94a0-e6ff8cabc8c3 None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:04:23.875062 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 670/1326] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:04:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:04:23.880184 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87caa2c4-b854-4c9f-81b5-c1698a8f8788 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:04:23.880590 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87caa2c4-b854-4c9f-81b5-c1698a8f8788 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:04:23.880738 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87caa2c4-b854-4c9f-81b5-c1698a8f8788 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:04:23.946613 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-87caa2c4-b854-4c9f-81b5-c1698a8f8788 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:04:23.947843 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-87caa2c4-b854-4c9f-81b5-c1698a8f8788 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:04:24.043444 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 657/1327] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:04:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:04:24.325888 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-321253a9-45a9-4442-94a0-e6ff8cabc8c3 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:04:24.327451 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-321253a9-45a9-4442-94a0-e6ff8cabc8c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:04:24.389650 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:04:24.434133 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-321253a9-45a9-4442-94a0-e6ff8cabc8c3 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:04:24.528269 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0a8f051-5ad4-4f87-b8fa-963dd527c9cc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:04:24.528878 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0a8f051-5ad4-4f87-b8fa-963dd527c9cc service neutron] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:04:24.529160 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0a8f051-5ad4-4f87-b8fa-963dd527c9cc service neutron] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:04:24.529429 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0a8f051-5ad4-4f87-b8fa-963dd527c9cc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:04:24.531045 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-d0a8f051-5ad4-4f87-b8fa-963dd527c9cc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:04:24.648776 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:04:24.651553 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0a8f051-5ad4-4f87-b8fa-963dd527c9cc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:04:24.653474 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0a8f051-5ad4-4f87-b8fa-963dd527c9cc service neutron] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:04:24.671820 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 671/1328] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:04:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 346 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:04:26.655407 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e185004-34b5-44da-b406-bb1906e15405 None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:04:26.655940 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e185004-34b5-44da-b406-bb1906e15405 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:04:26.655940 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e185004-34b5-44da-b406-bb1906e15405 None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:04:26.657327 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 658/1329] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:04:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:04:26.663258 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70d7f9da-b927-4b40-8cdb-0c2944550edb None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:04:26.663710 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70d7f9da-b927-4b40-8cdb-0c2944550edb None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:04:26.663810 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70d7f9da-b927-4b40-8cdb-0c2944550edb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:04:26.734119 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-70d7f9da-b927-4b40-8cdb-0c2944550edb None None] MFA Rules not processed for user `bb6b84c639b14608813a9a5e0dac6840`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:04:26.735360 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-70d7f9da-b927-4b40-8cdb-0c2944550edb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:04:26.830828 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 672/1330] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:04:26 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:04:27.113007 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e185004-34b5-44da-b406-bb1906e15405 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:04:27.114583 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-7e185004-34b5-44da-b406-bb1906e15405 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:04:27.151655 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:04:27.227269 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e185004-34b5-44da-b406-bb1906e15405 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:04:27.298319 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d8038c6-0a42-4c05-a647-90a076ca1039 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:04:27.298951 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d8038c6-0a42-4c05-a647-90a076ca1039 service neutron] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:04:27.299229 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d8038c6-0a42-4c05-a647-90a076ca1039 service neutron] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:04:27.299998 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d8038c6-0a42-4c05-a647-90a076ca1039 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:04:27.301560 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-2d8038c6-0a42-4c05-a647-90a076ca1039 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:04:27.378345 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:04:27.412131 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d8038c6-0a42-4c05-a647-90a076ca1039 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:04:27.413560 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d8038c6-0a42-4c05-a647-90a076ca1039 service neutron] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:04:27.432037 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 659/1331] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:04:27 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 320 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:04:29.572455 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a12b46b-4204-4e12-ae68-0b31c4da00cf None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:04:29.572977 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a12b46b-4204-4e12-ae68-0b31c4da00cf None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:04:29.573070 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a12b46b-4204-4e12-ae68-0b31c4da00cf None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:04:29.574489 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 673/1332] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:04:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:04:29.579425 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2d788d7-8071-4ccf-a5ac-3e4b3591d73d None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:04:29.579834 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2d788d7-8071-4ccf-a5ac-3e4b3591d73d None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:04:29.579970 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2d788d7-8071-4ccf-a5ac-3e4b3591d73d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:04:29.648331 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-a2d788d7-8071-4ccf-a5ac-3e4b3591d73d None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:04:29.649555 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-a2d788d7-8071-4ccf-a5ac-3e4b3591d73d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:04:29.745447 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 660/1333] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:04:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:04:30.027416 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a12b46b-4204-4e12-ae68-0b31c4da00cf None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:04:30.028951 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-0a12b46b-4204-4e12-ae68-0b31c4da00cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:04:30.098095 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:04:30.133000 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a12b46b-4204-4e12-ae68-0b31c4da00cf None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:04:30.203242 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3f9b3f8-164b-4d71-ae9e-1fa4d47d0dd3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:04:30.203833 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3f9b3f8-164b-4d71-ae9e-1fa4d47d0dd3 service neutron] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:04:30.204107 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3f9b3f8-164b-4d71-ae9e-1fa4d47d0dd3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:04:30.204373 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3f9b3f8-164b-4d71-ae9e-1fa4d47d0dd3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:04:30.205982 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-f3f9b3f8-164b-4d71-ae9e-1fa4d47d0dd3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:04:30.310638 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3f9b3f8-164b-4d71-ae9e-1fa4d47d0dd3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:04:30.312077 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3f9b3f8-164b-4d71-ae9e-1fa4d47d0dd3 service neutron] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:04:30.322329 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:04:30.330805 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 674/1334] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:04:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 304 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:04:32.345192 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0687d4e3-2cb9-4bf4-b34a-bd682612f780 None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:04:32.345720 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0687d4e3-2cb9-4bf4-b34a-bd682612f780 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:04:32.345720 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0687d4e3-2cb9-4bf4-b34a-bd682612f780 None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:04:32.347720 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 661/1335] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:04:32 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:04:32.353363 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d026a917-a249-41ce-aa7b-951722750b67 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:04:32.353813 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d026a917-a249-41ce-aa7b-951722750b67 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:04:32.353911 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d026a917-a249-41ce-aa7b-951722750b67 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:04:32.419273 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-d026a917-a249-41ce-aa7b-951722750b67 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:04:32.420480 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-d026a917-a249-41ce-aa7b-951722750b67 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:04:32.519822 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 675/1336] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:04:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:04:32.805412 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0687d4e3-2cb9-4bf4-b34a-bd682612f780 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:04:32.807003 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-0687d4e3-2cb9-4bf4-b34a-bd682612f780 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:04:32.845701 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:04:32.909410 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0687d4e3-2cb9-4bf4-b34a-bd682612f780 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:04:32.985829 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47a166a1-be75-4be5-9a58-8c44200abf48 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:04:32.986426 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47a166a1-be75-4be5-9a58-8c44200abf48 service neutron] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:04:32.987290 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47a166a1-be75-4be5-9a58-8c44200abf48 service neutron] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:04:32.987563 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47a166a1-be75-4be5-9a58-8c44200abf48 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:04:32.989133 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-47a166a1-be75-4be5-9a58-8c44200abf48 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:04:33.076002 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:04:33.103631 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47a166a1-be75-4be5-9a58-8c44200abf48 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:04:33.105067 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47a166a1-be75-4be5-9a58-8c44200abf48 service neutron] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:04:33.122738 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 662/1337] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:04:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 318 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:04:35.423508 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e03bac4-5cb8-4ee3-b4e3-a7e8154b53d3 None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:04:35.424568 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e03bac4-5cb8-4ee3-b4e3-a7e8154b53d3 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:04:35.424568 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e03bac4-5cb8-4ee3-b4e3-a7e8154b53d3 None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:04:35.425924 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 676/1338] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:04:35 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:04:35.431433 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c29de2e-87bc-4e2e-8e7f-2d70b1d2c3cf None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:04:35.431882 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c29de2e-87bc-4e2e-8e7f-2d70b1d2c3cf None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:04:35.431997 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c29de2e-87bc-4e2e-8e7f-2d70b1d2c3cf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:04:35.505437 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-4c29de2e-87bc-4e2e-8e7f-2d70b1d2c3cf None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:04:35.506671 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-4c29de2e-87bc-4e2e-8e7f-2d70b1d2c3cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:04:35.601106 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 663/1339] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:04:35 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:04:35.882377 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e03bac4-5cb8-4ee3-b4e3-a7e8154b53d3 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:04:35.883951 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-8e03bac4-5cb8-4ee3-b4e3-a7e8154b53d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:04:35.961209 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:04:35.989938 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e03bac4-5cb8-4ee3-b4e3-a7e8154b53d3 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:04:36.061206 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93105845-ea18-422d-af51-7607fff80210 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:04:36.061792 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93105845-ea18-422d-af51-7607fff80210 service neutron] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:04:36.062073 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93105845-ea18-422d-af51-7607fff80210 service neutron] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:04:36.062671 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93105845-ea18-422d-af51-7607fff80210 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:04:36.064265 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-93105845-ea18-422d-af51-7607fff80210 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:04:36.171368 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93105845-ea18-422d-af51-7607fff80210 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:04:36.172736 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93105845-ea18-422d-af51-7607fff80210 service neutron] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:04:36.191577 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 677/1340] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:04:35 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:04:39.978958 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-923b15b8-e983-4c87-a83f-d1738adbaee4 None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:04:39.979490 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-923b15b8-e983-4c87-a83f-d1738adbaee4 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:04:39.979490 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-923b15b8-e983-4c87-a83f-d1738adbaee4 None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:04:39.980855 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 664/1341] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:04:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:04:39.985954 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a9b7bf8-ce7f-4410-8714-db9e787d688e None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:04:39.986362 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a9b7bf8-ce7f-4410-8714-db9e787d688e None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:04:39.987054 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a9b7bf8-ce7f-4410-8714-db9e787d688e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:04:39.992486 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:04:40.054371 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-4a9b7bf8-ce7f-4410-8714-db9e787d688e None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:04:40.055604 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-4a9b7bf8-ce7f-4410-8714-db9e787d688e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:04:40.151766 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 678/1342] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:04:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:04:40.437521 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-923b15b8-e983-4c87-a83f-d1738adbaee4 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:04:40.439073 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-923b15b8-e983-4c87-a83f-d1738adbaee4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:04:40.491394 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:04:40.547568 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-923b15b8-e983-4c87-a83f-d1738adbaee4 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:04:40.619457 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b56b516-8539-490a-982e-8700de8f94d4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:04:40.620052 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b56b516-8539-490a-982e-8700de8f94d4 service neutron] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:04:40.620325 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b56b516-8539-490a-982e-8700de8f94d4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:04:40.620590 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b56b516-8539-490a-982e-8700de8f94d4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:04:40.622148 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-3b56b516-8539-490a-982e-8700de8f94d4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:04:40.714388 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:04:40.731145 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b56b516-8539-490a-982e-8700de8f94d4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:04:40.732581 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b56b516-8539-490a-982e-8700de8f94d4 service neutron] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:04:40.750153 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 665/1343] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:04:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:04:43.287609 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b66d5e-c9b7-4205-bc5f-0c3a377683eb None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:04:43.288188 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b66d5e-c9b7-4205-bc5f-0c3a377683eb None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:04:43.288188 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b66d5e-c9b7-4205-bc5f-0c3a377683eb None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:04:43.289664 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 679/1344] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:04:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:04:43.295077 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e613964c-ca64-4279-9fb1-559949b96f32 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:04:43.295510 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e613964c-ca64-4279-9fb1-559949b96f32 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:04:43.295611 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e613964c-ca64-4279-9fb1-559949b96f32 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:04:43.364997 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-e613964c-ca64-4279-9fb1-559949b96f32 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:04:43.366208 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-e613964c-ca64-4279-9fb1-559949b96f32 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:04:43.466791 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 666/1345] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:04:43 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:04:43.749608 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37b66d5e-c9b7-4205-bc5f-0c3a377683eb None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:04:43.751174 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-37b66d5e-c9b7-4205-bc5f-0c3a377683eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:04:43.838950 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:04:43.857694 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37b66d5e-c9b7-4205-bc5f-0c3a377683eb None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:04:43.936894 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e121db5e-3713-4e73-aee9-117f42551bbe service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:04:43.937544 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e121db5e-3713-4e73-aee9-117f42551bbe service neutron] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:04:43.937825 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e121db5e-3713-4e73-aee9-117f42551bbe service neutron] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:04:43.938124 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e121db5e-3713-4e73-aee9-117f42551bbe service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:04:43.939702 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-e121db5e-3713-4e73-aee9-117f42551bbe service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:04:44.055228 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e121db5e-3713-4e73-aee9-117f42551bbe service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:04:44.056661 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e121db5e-3713-4e73-aee9-117f42551bbe service neutron] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:04:44.074372 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 680/1346] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:04:43 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 325 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:04:47.894214 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478c5fec-f610-4936-9043-5e7cf13cd48b None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:04:47.894700 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478c5fec-f610-4936-9043-5e7cf13cd48b None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:04:47.894794 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478c5fec-f610-4936-9043-5e7cf13cd48b None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:04:47.896203 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 667/1347] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:04:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:04:47.901569 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa8b34af-a3d9-48ea-9588-e52c269e5f82 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:04:47.901993 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa8b34af-a3d9-48ea-9588-e52c269e5f82 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:04:47.902103 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa8b34af-a3d9-48ea-9588-e52c269e5f82 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:04:47.918279 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:04:47.984464 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-aa8b34af-a3d9-48ea-9588-e52c269e5f82 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:04:47.985689 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-aa8b34af-a3d9-48ea-9588-e52c269e5f82 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:04:48.092310 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 681/1348] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:04:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:04:48.373901 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-478c5fec-f610-4936-9043-5e7cf13cd48b None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:04:48.375482 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-478c5fec-f610-4936-9043-5e7cf13cd48b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:04:48.436481 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:04:48.485847 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-478c5fec-f610-4936-9043-5e7cf13cd48b None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:04:48.560009 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ce9bda4-b09d-4e29-b936-5695fae039ab service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:04:48.561334 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ce9bda4-b09d-4e29-b936-5695fae039ab service neutron] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:04:48.561334 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ce9bda4-b09d-4e29-b936-5695fae039ab service neutron] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:04:48.561334 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ce9bda4-b09d-4e29-b936-5695fae039ab service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:04:48.562750 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-7ce9bda4-b09d-4e29-b936-5695fae039ab service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:04:48.671939 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:04:48.676378 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ce9bda4-b09d-4e29-b936-5695fae039ab service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:04:48.677824 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ce9bda4-b09d-4e29-b936-5695fae039ab service neutron] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:04:48.695870 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 668/1349] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:04:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 323 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:04:53.531625 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09661892-1650-43d1-a906-2edf2ac10039 None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:04:53.532417 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09661892-1650-43d1-a906-2edf2ac10039 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:04:53.532849 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09661892-1650-43d1-a906-2edf2ac10039 None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:04:53.534606 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 682/1350] 10.222.0.77 () {54 vars in 921 bytes} [Tue Aug 30 05:04:53 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:04:53.542967 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaac9271-9d02-4cc3-ae1d-bd3982e7e9f4 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:04:53.542967 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaac9271-9d02-4cc3-ae1d-bd3982e7e9f4 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:04:53.542967 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaac9271-9d02-4cc3-ae1d-bd3982e7e9f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:04:53.698730 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-eaac9271-9d02-4cc3-ae1d-bd3982e7e9f4 None None] MFA Rules not processed for user `1972959402704b89925001cd8d90cb0e`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:04:53.699938 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-eaac9271-9d02-4cc3-ae1d-bd3982e7e9f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:04:53.818980 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 669/1351] 10.222.0.77 () {58 vars in 1018 bytes} [Tue Aug 30 05:04:53 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 279 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:04:57.817476 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29431f01-242c-4788-8e93-67dd0205aa46 None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:04:57.818061 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29431f01-242c-4788-8e93-67dd0205aa46 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:04:57.818061 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29431f01-242c-4788-8e93-67dd0205aa46 None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:04:57.819380 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 683/1352] 10.222.0.77 () {54 vars in 921 bytes} [Tue Aug 30 05:04:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:04:57.824469 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a3144d9-109e-4020-9818-604702e1ea2e None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:04:57.824890 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a3144d9-109e-4020-9818-604702e1ea2e None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:04:57.825059 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a3144d9-109e-4020-9818-604702e1ea2e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:04:57.882190 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:04:57.912725 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-5a3144d9-109e-4020-9818-604702e1ea2e None None] MFA Rules not processed for user `1972959402704b89925001cd8d90cb0e`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:04:57.914036 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-5a3144d9-109e-4020-9818-604702e1ea2e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:04:58.027712 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 670/1353] 10.222.0.77 () {58 vars in 1018 bytes} [Tue Aug 30 05:04:57 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 204 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:04:58.090489 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=59 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:04:59.630970 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-026d4053-1441-4e66-b991-41ce26667c98 None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:04:59.631620 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-026d4053-1441-4e66-b991-41ce26667c98 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:04:59.631620 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-026d4053-1441-4e66-b991-41ce26667c98 None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:04:59.632826 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 684/1354] 10.222.0.77 () {54 vars in 921 bytes} [Tue Aug 30 05:04:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:04:59.640366 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55144f7e-54b3-477d-9548-91f2d4284e70 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:04:59.640795 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55144f7e-54b3-477d-9548-91f2d4284e70 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:04:59.640953 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55144f7e-54b3-477d-9548-91f2d4284e70 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:04:59.715969 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-55144f7e-54b3-477d-9548-91f2d4284e70 None None] MFA Rules not processed for user `1972959402704b89925001cd8d90cb0e`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:04:59.717073 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-55144f7e-54b3-477d-9548-91f2d4284e70 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:04:59.758416 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:04:59.825296 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 671/1355] 10.222.0.77 () {58 vars in 1018 bytes} [Tue Aug 30 05:04:59 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:05:03.016181 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2daf1a9e-68c0-47ba-bbd9-751873bcf52c None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:05:03.016693 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2daf1a9e-68c0-47ba-bbd9-751873bcf52c None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:05:03.016801 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2daf1a9e-68c0-47ba-bbd9-751873bcf52c None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:05:03.018591 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 685/1356] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:05:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:05:03.023898 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e558976e-8504-4170-a127-84ad2b81ce8f None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:05:03.024313 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e558976e-8504-4170-a127-84ad2b81ce8f None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:05:03.024455 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e558976e-8504-4170-a127-84ad2b81ce8f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:05:03.110587 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-e558976e-8504-4170-a127-84ad2b81ce8f None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:05:03.113558 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-e558976e-8504-4170-a127-84ad2b81ce8f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:05:03.228964 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 672/1357] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:05:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 206 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 05:05:03.532052 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eebc0db5-5967-4ccb-8815-d21a4e9b0b82 None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:05:03.532651 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eebc0db5-5967-4ccb-8815-d21a4e9b0b82 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:05:03.533738 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eebc0db5-5967-4ccb-8815-d21a4e9b0b82 None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:05:03.535385 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 686/1358] 10.222.0.77 () {54 vars in 962 bytes} [Tue Aug 30 05:05:03 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:05:03.544338 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dde6316e-049a-4a5e-a5b1-1338c559b5f2 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:05:03.544955 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dde6316e-049a-4a5e-a5b1-1338c559b5f2 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:05:03.545449 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dde6316e-049a-4a5e-a5b1-1338c559b5f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:05:03.586164 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:05:03.653202 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-dde6316e-049a-4a5e-a5b1-1338c559b5f2 None None] MFA Rules not processed for user `886eb5e00bbf4ec292eb962b665b37d0`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:05:03.654388 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-dde6316e-049a-4a5e-a5b1-1338c559b5f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:05:03.776265 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 673/1359] 10.222.0.77 () {58 vars in 1059 bytes} [Tue Aug 30 05:05:03 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 234 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 05:05:03.784481 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eebc0db5-5967-4ccb-8815-d21a4e9b0b82 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:05:03.786255 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-eebc0db5-5967-4ccb-8815-d21a4e9b0b82 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:05:03.943048 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eebc0db5-5967-4ccb-8815-d21a4e9b0b82 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:05:04.024705 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:05:04.070159 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-681a2eef-fc2a-4766-b5a8-e8b264fe47b9 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '886eb5e00bbf4ec292eb962b665b37d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:05:04.070755 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-681a2eef-fc2a-4766-b5a8-e8b264fe47b9 service nova] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:05:04.071058 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-681a2eef-fc2a-4766-b5a8-e8b264fe47b9 service nova] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:05:04.071448 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-681a2eef-fc2a-4766-b5a8-e8b264fe47b9 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:05:04.072914 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-681a2eef-fc2a-4766-b5a8-e8b264fe47b9 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:05:04.165736 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-681a2eef-fc2a-4766-b5a8-e8b264fe47b9 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:05:04.167753 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-681a2eef-fc2a-4766-b5a8-e8b264fe47b9 service nova] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:05:04.187444 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 687/1360] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:05:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 403 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:05:06.321195 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf994de-d001-4d0c-aa94-9ca97911fb9b None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:05:06.321737 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf994de-d001-4d0c-aa94-9ca97911fb9b None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:05:06.321737 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf994de-d001-4d0c-aa94-9ca97911fb9b None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:05:06.323298 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 674/1361] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:05:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:05:06.328213 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f1278ac-2487-411f-a6b0-793bc11ce453 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:05:06.328635 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f1278ac-2487-411f-a6b0-793bc11ce453 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:05:06.328757 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f1278ac-2487-411f-a6b0-793bc11ce453 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:05:06.398198 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-6f1278ac-2487-411f-a6b0-793bc11ce453 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:05:06.399416 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-6f1278ac-2487-411f-a6b0-793bc11ce453 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:05:06.439316 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:05:06.517577 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 688/1362] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:05:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:05:06.807036 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdf994de-d001-4d0c-aa94-9ca97911fb9b None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:05:06.808655 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-bdf994de-d001-4d0c-aa94-9ca97911fb9b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:05:06.878433 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:05:06.968287 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdf994de-d001-4d0c-aa94-9ca97911fb9b None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:05:07.087685 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abab9407-ba11-466f-aa2e-d1a8b60d310e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '886eb5e00bbf4ec292eb962b665b37d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:05:07.088391 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abab9407-ba11-466f-aa2e-d1a8b60d310e service nova] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:05:07.088767 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abab9407-ba11-466f-aa2e-d1a8b60d310e service nova] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:05:07.089135 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abab9407-ba11-466f-aa2e-d1a8b60d310e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:05:07.090811 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-abab9407-ba11-466f-aa2e-d1a8b60d310e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:05:07.125382 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:05:07.208795 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abab9407-ba11-466f-aa2e-d1a8b60d310e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:05:07.210417 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abab9407-ba11-466f-aa2e-d1a8b60d310e service nova] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:05:07.229989 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 675/1363] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:05:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 423 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:05:09.510298 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e15f5bc8-ae5c-4e23-a455-743b75aa43ec None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:05:09.511185 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e15f5bc8-ae5c-4e23-a455-743b75aa43ec None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:05:09.511616 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e15f5bc8-ae5c-4e23-a455-743b75aa43ec None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:05:09.513115 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 689/1364] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:05:09 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:05:09.519083 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd6f74ee-c01d-4de8-9dd2-bdecb9e4cbc2 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:05:09.519838 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd6f74ee-c01d-4de8-9dd2-bdecb9e4cbc2 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:05:09.520284 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd6f74ee-c01d-4de8-9dd2-bdecb9e4cbc2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:05:09.590356 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-dd6f74ee-c01d-4de8-9dd2-bdecb9e4cbc2 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:05:09.591647 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-dd6f74ee-c01d-4de8-9dd2-bdecb9e4cbc2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:05:09.691692 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:05:09.698906 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 676/1365] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:05:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:05:09.988166 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd2b3788-fe6c-4171-a7da-7406763a203f None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:05:09.988464 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd2b3788-fe6c-4171-a7da-7406763a203f None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:05:09.988722 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd2b3788-fe6c-4171-a7da-7406763a203f None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:05:09.990257 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 690/1366] 10.222.0.77 () {54 vars in 962 bytes} [Tue Aug 30 05:05:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:05:09.995854 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-912662df-0c1d-4bff-9521-34b280f11df8 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:05:09.996139 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-912662df-0c1d-4bff-9521-34b280f11df8 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:05:09.996410 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-912662df-0c1d-4bff-9521-34b280f11df8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:05:10.084620 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-912662df-0c1d-4bff-9521-34b280f11df8 None None] MFA Rules not processed for user `886eb5e00bbf4ec292eb962b665b37d0`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:05:10.085840 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-912662df-0c1d-4bff-9521-34b280f11df8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:05:10.211383 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 677/1367] 10.222.0.77 () {58 vars in 1059 bytes} [Tue Aug 30 05:05:09 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 216 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:05:10.220622 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd2b3788-fe6c-4171-a7da-7406763a203f None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:05:10.222242 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-dd2b3788-fe6c-4171-a7da-7406763a203f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:05:10.381554 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd2b3788-fe6c-4171-a7da-7406763a203f None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:05:10.402682 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:05:10.513433 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0fea3d1-4a18-42ac-bad9-66153932b24a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '886eb5e00bbf4ec292eb962b665b37d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:05:10.514112 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0fea3d1-4a18-42ac-bad9-66153932b24a service nova] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:05:10.514448 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0fea3d1-4a18-42ac-bad9-66153932b24a service nova] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:05:10.514791 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0fea3d1-4a18-42ac-bad9-66153932b24a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:05:10.516569 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-a0fea3d1-4a18-42ac-bad9-66153932b24a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:05:10.618418 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0fea3d1-4a18-42ac-bad9-66153932b24a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:05:10.620146 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0fea3d1-4a18-42ac-bad9-66153932b24a service nova] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:05:10.639034 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 691/1368] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:05:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 419 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:05:11.187887 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2410d0e7-ba3a-4f2f-934c-d786452d7ab3 None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:05:11.188441 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2410d0e7-ba3a-4f2f-934c-d786452d7ab3 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:05:11.188441 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2410d0e7-ba3a-4f2f-934c-d786452d7ab3 None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:05:11.190472 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 678/1369] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:05:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:05:11.195452 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1805e8f9-dd83-4613-bbd6-9e62a096c765 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:05:11.195741 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1805e8f9-dd83-4613-bbd6-9e62a096c765 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:05:11.196016 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1805e8f9-dd83-4613-bbd6-9e62a096c765 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:05:11.229035 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:05:11.265253 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-1805e8f9-dd83-4613-bbd6-9e62a096c765 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:05:11.266460 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-1805e8f9-dd83-4613-bbd6-9e62a096c765 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:05:11.365329 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 692/1370] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:05:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:05:11.950663 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-417ca4db-999c-447a-8772-b7d0921a1d1b None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:05:11.950663 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-417ca4db-999c-447a-8772-b7d0921a1d1b None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:05:11.950663 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-417ca4db-999c-447a-8772-b7d0921a1d1b None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:05:11.953226 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 679/1371] 10.222.0.77 () {54 vars in 964 bytes} [Tue Aug 30 05:05:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:05:11.956528 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e063d64-6384-416f-bab3-6e8daf04e3bc None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:05:11.957043 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e063d64-6384-416f-bab3-6e8daf04e3bc None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:05:11.957195 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e063d64-6384-416f-bab3-6e8daf04e3bc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:05:12.045783 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-3e063d64-6384-416f-bab3-6e8daf04e3bc None None] MFA Rules not processed for user `019a79b2affe478b8a30d1c538d0415c`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:05:12.047019 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-3e063d64-6384-416f-bab3-6e8daf04e3bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:05:12.091802 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:05:12.124427 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 693/1372] 10.222.0.77 () {58 vars in 1061 bytes} [Tue Aug 30 05:05:11 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 05:05:12.132227 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-417ca4db-999c-447a-8772-b7d0921a1d1b None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:05:12.134133 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-417ca4db-999c-447a-8772-b7d0921a1d1b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:05:12.186466 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:05:12.243783 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-417ca4db-999c-447a-8772-b7d0921a1d1b None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:05:12.324353 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dbe44f7-48d0-4d4f-959a-c13839057b2e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '019a79b2affe478b8a30d1c538d0415c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:05:12.324982 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dbe44f7-48d0-4d4f-959a-c13839057b2e service cinder] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:05:12.325305 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dbe44f7-48d0-4d4f-959a-c13839057b2e service cinder] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:05:12.325477 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dbe44f7-48d0-4d4f-959a-c13839057b2e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:05:12.327176 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-6dbe44f7-48d0-4d4f-959a-c13839057b2e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:05:12.435401 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:05:12.453853 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dbe44f7-48d0-4d4f-959a-c13839057b2e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:05:12.455946 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dbe44f7-48d0-4d4f-959a-c13839057b2e service cinder] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:05:12.477338 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 680/1373] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:05:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 345 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:05:12.609703 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f43ac460-abd9-475e-8bdd-6c9eefac531c None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:05:12.609997 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f43ac460-abd9-475e-8bdd-6c9eefac531c None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:05:12.610273 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f43ac460-abd9-475e-8bdd-6c9eefac531c None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:05:12.613284 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 694/1374] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:05:12 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:05:12.617195 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80eb5bf4-e155-4dc0-a8bc-e8baadc30d16 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:05:12.617472 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80eb5bf4-e155-4dc0-a8bc-e8baadc30d16 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:05:12.618227 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80eb5bf4-e155-4dc0-a8bc-e8baadc30d16 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:05:12.711648 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-80eb5bf4-e155-4dc0-a8bc-e8baadc30d16 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:05:12.712807 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-80eb5bf4-e155-4dc0-a8bc-e8baadc30d16 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:05:12.809553 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 681/1375] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:05:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:05:13.097978 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f43ac460-abd9-475e-8bdd-6c9eefac531c None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:05:13.099583 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-f43ac460-abd9-475e-8bdd-6c9eefac531c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:05:13.253176 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f43ac460-abd9-475e-8bdd-6c9eefac531c None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:05:13.305685 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:05:13.373235 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5267adc-71df-4715-aad2-edd36843c701 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '886eb5e00bbf4ec292eb962b665b37d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:05:13.373748 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5267adc-71df-4715-aad2-edd36843c701 service nova] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:05:13.374023 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5267adc-71df-4715-aad2-edd36843c701 service nova] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:05:13.374289 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5267adc-71df-4715-aad2-edd36843c701 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:05:13.375919 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-b5267adc-71df-4715-aad2-edd36843c701 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:05:13.469143 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5267adc-71df-4715-aad2-edd36843c701 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:05:13.471279 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5267adc-71df-4715-aad2-edd36843c701 service nova] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:05:13.489759 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 695/1376] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:05:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 392 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:05:15.415491 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49e9b212-5fac-4e8d-9f8a-995b2f417c31 None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:05:15.416067 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49e9b212-5fac-4e8d-9f8a-995b2f417c31 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:05:15.416067 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49e9b212-5fac-4e8d-9f8a-995b2f417c31 None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:05:15.417439 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 682/1377] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:05:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:05:15.422950 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35c67182-3e3e-4ebd-8b52-697216c78e56 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:05:15.423390 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35c67182-3e3e-4ebd-8b52-697216c78e56 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:05:15.423522 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35c67182-3e3e-4ebd-8b52-697216c78e56 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:05:15.511984 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-35c67182-3e3e-4ebd-8b52-697216c78e56 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:05:15.513110 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-35c67182-3e3e-4ebd-8b52-697216c78e56 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:05:15.522050 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:05:15.613502 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 696/1378] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:05:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 05:05:15.904086 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49e9b212-5fac-4e8d-9f8a-995b2f417c31 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:05:15.905597 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-49e9b212-5fac-4e8d-9f8a-995b2f417c31 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:05:15.999389 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:05:16.120908 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49e9b212-5fac-4e8d-9f8a-995b2f417c31 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:05:16.246519 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2df8161-4ee6-4d5b-9dbb-f40024f03c92 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '886eb5e00bbf4ec292eb962b665b37d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:05:16.247254 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2df8161-4ee6-4d5b-9dbb-f40024f03c92 service nova] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:05:16.247612 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2df8161-4ee6-4d5b-9dbb-f40024f03c92 service nova] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:05:16.247961 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2df8161-4ee6-4d5b-9dbb-f40024f03c92 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:05:16.249620 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-c2df8161-4ee6-4d5b-9dbb-f40024f03c92 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:05:16.269122 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:05:16.345940 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2df8161-4ee6-4d5b-9dbb-f40024f03c92 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:05:16.347568 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2df8161-4ee6-4d5b-9dbb-f40024f03c92 service nova] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:05:16.367154 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 683/1379] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:05:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 463 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:05:18.310271 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caf89c05-aeb2-4f48-b5ba-5b461e22ba43 None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:05:18.310789 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caf89c05-aeb2-4f48-b5ba-5b461e22ba43 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:05:18.310789 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caf89c05-aeb2-4f48-b5ba-5b461e22ba43 None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:05:18.316759 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 697/1380] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:05:18 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:05:18.318382 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd507e8-f1b3-4f37-9c00-74ace692cbee None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:05:18.319403 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd507e8-f1b3-4f37-9c00-74ace692cbee None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:05:18.319937 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd507e8-f1b3-4f37-9c00-74ace692cbee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:05:18.392773 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-efd507e8-f1b3-4f37-9c00-74ace692cbee None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:05:18.393983 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-efd507e8-f1b3-4f37-9c00-74ace692cbee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:05:18.499844 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:05:18.507622 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 684/1381] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:05:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:05:18.799745 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caf89c05-aeb2-4f48-b5ba-5b461e22ba43 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:05:18.801266 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-caf89c05-aeb2-4f48-b5ba-5b461e22ba43 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:05:18.949699 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:05:18.961287 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caf89c05-aeb2-4f48-b5ba-5b461e22ba43 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:05:19.081539 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fae9936b-a23d-45d9-9f4b-372236ef9f84 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '886eb5e00bbf4ec292eb962b665b37d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:05:19.082132 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae9936b-a23d-45d9-9f4b-372236ef9f84 service nova] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:05:19.082405 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae9936b-a23d-45d9-9f4b-372236ef9f84 service nova] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:05:19.082670 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae9936b-a23d-45d9-9f4b-372236ef9f84 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:05:19.084272 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-fae9936b-a23d-45d9-9f4b-372236ef9f84 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:05:19.182426 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fae9936b-a23d-45d9-9f4b-372236ef9f84 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:05:19.184076 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fae9936b-a23d-45d9-9f4b-372236ef9f84 service nova] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:05:19.196292 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:05:19.205863 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 698/1382] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:05:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 406 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:05:21.152734 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed27658-f53d-4ed6-9617-3718f71051c0 None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:05:21.153318 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed27658-f53d-4ed6-9617-3718f71051c0 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:05:21.153318 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed27658-f53d-4ed6-9617-3718f71051c0 None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:05:21.154637 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 685/1383] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:05:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:05:21.160520 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4e51e5c-698e-415f-805a-ceee9b54cd09 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:05:21.160968 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4e51e5c-698e-415f-805a-ceee9b54cd09 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:05:21.161068 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4e51e5c-698e-415f-805a-ceee9b54cd09 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:05:21.238628 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-c4e51e5c-698e-415f-805a-ceee9b54cd09 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:05:21.240296 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-c4e51e5c-698e-415f-805a-ceee9b54cd09 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:05:21.354034 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 699/1384] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:05:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:05:21.639917 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ed27658-f53d-4ed6-9617-3718f71051c0 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:05:21.641436 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-5ed27658-f53d-4ed6-9617-3718f71051c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:05:21.794598 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ed27658-f53d-4ed6-9617-3718f71051c0 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:05:21.868342 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:05:21.929422 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-980f4ba4-37eb-42db-9298-7a7b2e5b0190 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '886eb5e00bbf4ec292eb962b665b37d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:05:21.930046 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-980f4ba4-37eb-42db-9298-7a7b2e5b0190 service nova] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:05:21.930325 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-980f4ba4-37eb-42db-9298-7a7b2e5b0190 service nova] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:05:21.930594 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-980f4ba4-37eb-42db-9298-7a7b2e5b0190 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:05:21.932277 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-980f4ba4-37eb-42db-9298-7a7b2e5b0190 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:05:22.021707 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-980f4ba4-37eb-42db-9298-7a7b2e5b0190 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:05:22.023327 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-980f4ba4-37eb-42db-9298-7a7b2e5b0190 service nova] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:05:22.041318 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 686/1385] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:05:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 402 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:05:24.366485 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d4ae44b-48eb-4b69-9e17-4cc946c28e90 None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:05:24.367050 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d4ae44b-48eb-4b69-9e17-4cc946c28e90 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:05:24.367050 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d4ae44b-48eb-4b69-9e17-4cc946c28e90 None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:05:24.368398 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 700/1386] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:05:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:05:24.374765 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccace89b-e566-41cc-b565-d16bbccac33b None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:05:24.375220 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccace89b-e566-41cc-b565-d16bbccac33b None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:05:24.375347 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccace89b-e566-41cc-b565-d16bbccac33b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:05:24.449808 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-ccace89b-e566-41cc-b565-d16bbccac33b None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:05:24.451054 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-ccace89b-e566-41cc-b565-d16bbccac33b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:05:24.486039 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:05:24.551021 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 687/1387] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:05:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:05:24.839969 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d4ae44b-48eb-4b69-9e17-4cc946c28e90 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:05:24.841490 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-1d4ae44b-48eb-4b69-9e17-4cc946c28e90 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:05:24.919799 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:05:25.001203 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d4ae44b-48eb-4b69-9e17-4cc946c28e90 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:05:25.117250 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-238456a7-9469-4786-b6fc-8eacaff344b2 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '886eb5e00bbf4ec292eb962b665b37d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:05:25.117799 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-238456a7-9469-4786-b6fc-8eacaff344b2 service nova] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:05:25.118078 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-238456a7-9469-4786-b6fc-8eacaff344b2 service nova] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:05:25.118342 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-238456a7-9469-4786-b6fc-8eacaff344b2 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:05:25.120000 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-238456a7-9469-4786-b6fc-8eacaff344b2 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:05:25.158196 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:05:25.225626 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-238456a7-9469-4786-b6fc-8eacaff344b2 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:05:25.227299 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-238456a7-9469-4786-b6fc-8eacaff344b2 service nova] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:05:25.245917 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 701/1388] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:05:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 406 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:05:27.184771 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f53733e-e1b2-45da-8e50-7285ad5b74e3 None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:05:27.185327 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f53733e-e1b2-45da-8e50-7285ad5b74e3 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:05:27.185327 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f53733e-e1b2-45da-8e50-7285ad5b74e3 None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:05:27.186855 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 688/1389] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:05:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:05:27.192491 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b42bd22d-a693-439b-9569-6b875c796bde None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:05:27.192952 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b42bd22d-a693-439b-9569-6b875c796bde None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:05:27.193042 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b42bd22d-a693-439b-9569-6b875c796bde None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:05:27.258231 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-b42bd22d-a693-439b-9569-6b875c796bde None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:05:27.259452 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-b42bd22d-a693-439b-9569-6b875c796bde None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:05:27.353668 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 702/1390] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:05:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:05:27.437740 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d763de-5995-4a98-bb83-56adc70a33e8 None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:05:27.437957 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d763de-5995-4a98-bb83-56adc70a33e8 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:05:27.438234 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d763de-5995-4a98-bb83-56adc70a33e8 None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:05:27.439750 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 689/1391] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:05:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:05:27.445062 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60543c15-d0ef-459b-b834-b5dee5051c51 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:05:27.445348 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60543c15-d0ef-459b-b834-b5dee5051c51 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:05:27.445747 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60543c15-d0ef-459b-b834-b5dee5051c51 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:05:27.464597 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:05:27.516529 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-60543c15-d0ef-459b-b834-b5dee5051c51 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:05:27.517740 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-60543c15-d0ef-459b-b834-b5dee5051c51 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:05:27.616114 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 703/1392] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:05:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:05:27.639019 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78d763de-5995-4a98-bb83-56adc70a33e8 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:05:27.640600 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-78d763de-5995-4a98-bb83-56adc70a33e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:05:27.785155 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78d763de-5995-4a98-bb83-56adc70a33e8 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:05:27.801329 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:05:27.908903 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a95e072-bd8c-404d-8aa5-ad81bcee6b56 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '886eb5e00bbf4ec292eb962b665b37d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:05:27.909508 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a95e072-bd8c-404d-8aa5-ad81bcee6b56 service nova] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:05:27.909782 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a95e072-bd8c-404d-8aa5-ad81bcee6b56 service nova] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:05:27.910049 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a95e072-bd8c-404d-8aa5-ad81bcee6b56 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:05:27.911757 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-6a95e072-bd8c-404d-8aa5-ad81bcee6b56 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:05:27.922270 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9a80f5b-8759-4d7f-8bbe-0cc855857cc9 None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:05:27.922555 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9a80f5b-8759-4d7f-8bbe-0cc855857cc9 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:05:27.922887 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9a80f5b-8759-4d7f-8bbe-0cc855857cc9 None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:05:27.924435 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 704/1393] 10.222.0.77 () {54 vars in 967 bytes} [Tue Aug 30 05:05:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 05:05:27.929186 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-309b83a5-1c98-457a-aa11-b5a5b7f381d4 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:05:27.929466 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-309b83a5-1c98-457a-aa11-b5a5b7f381d4 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:05:27.929735 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-309b83a5-1c98-457a-aa11-b5a5b7f381d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:05:28.002963 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a95e072-bd8c-404d-8aa5-ad81bcee6b56 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:05:28.004556 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a95e072-bd8c-404d-8aa5-ad81bcee6b56 service nova] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:05:28.006669 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-309b83a5-1c98-457a-aa11-b5a5b7f381d4 None None] MFA Rules not processed for user `e0a002ac0bf240a891ac7304178d3f0e`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:05:28.007902 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-309b83a5-1c98-457a-aa11-b5a5b7f381d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:05:28.022974 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 690/1394] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:05:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 384 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:05:28.044493 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:05:28.095533 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 705/1395] 10.222.0.77 () {58 vars in 1064 bytes} [Tue Aug 30 05:05:27 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 05:05:28.103878 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a95e072-bd8c-404d-8aa5-ad81bcee6b56 service nova] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:05:28.105106 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-6a95e072-bd8c-404d-8aa5-ad81bcee6b56 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:05:28.130181 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:05:28.187617 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a95e072-bd8c-404d-8aa5-ad81bcee6b56 service nova] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:05:28.264241 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25d98ef0-b852-48d3-9d68-1e0726a44ac5 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a002ac0bf240a891ac7304178d3f0e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:05:28.264839 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d98ef0-b852-48d3-9d68-1e0726a44ac5 service glance] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:05:28.265114 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d98ef0-b852-48d3-9d68-1e0726a44ac5 service glance] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:05:28.265381 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d98ef0-b852-48d3-9d68-1e0726a44ac5 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:05:28.267527 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-25d98ef0-b852-48d3-9d68-1e0726a44ac5 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:05:28.363337 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:05:28.375479 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25d98ef0-b852-48d3-9d68-1e0726a44ac5 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:05:28.377441 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25d98ef0-b852-48d3-9d68-1e0726a44ac5 service glance] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:05:28.395963 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 691/1396] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:05:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 293 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:05:29.121528 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ac0b52-c6b4-46fb-b929-8ee1579b028b None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:05:29.122005 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ac0b52-c6b4-46fb-b929-8ee1579b028b None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:05:29.122182 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ac0b52-c6b4-46fb-b929-8ee1579b028b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:05:29.190842 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-69ac0b52-c6b4-46fb-b929-8ee1579b028b None None] MFA Rules not processed for user `fe51dbec91624ed1ab00b441e29a3dd5`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:05:29.192037 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-69ac0b52-c6b4-46fb-b929-8ee1579b028b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:05:29.277181 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 706/1397] 10.222.0.77 () {58 vars in 1009 bytes} [Tue Aug 30 05:05:29 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:05:29.318299 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac03b40d-43dc-4b01-8099-04ca75fddba9 None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:05:29.318850 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac03b40d-43dc-4b01-8099-04ca75fddba9 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:05:29.318948 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac03b40d-43dc-4b01-8099-04ca75fddba9 None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:05:29.320236 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 692/1398] 10.222.0.77 () {54 vars in 943 bytes} [Tue Aug 30 05:05:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:05:29.326752 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-658ce335-6d40-49bc-8cc0-bfa37510c30b None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:05:29.327179 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-658ce335-6d40-49bc-8cc0-bfa37510c30b None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:05:29.327458 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-658ce335-6d40-49bc-8cc0-bfa37510c30b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:05:29.384823 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:05:29.403415 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-658ce335-6d40-49bc-8cc0-bfa37510c30b None None] MFA Rules not processed for user `e0a7f7afc3d44ec788c05d005e52a1b8`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:05:29.404614 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-658ce335-6d40-49bc-8cc0-bfa37510c30b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:05:29.518834 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 707/1399] 10.222.0.77 () {58 vars in 1040 bytes} [Tue Aug 30 05:05:29 2022] POST /identity/v3/auth/tokens => generated 3304 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:05:29.528013 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac03b40d-43dc-4b01-8099-04ca75fddba9 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:05:29.529552 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-ac03b40d-43dc-4b01-8099-04ca75fddba9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:05:29.668606 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac03b40d-43dc-4b01-8099-04ca75fddba9 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:05:29.732640 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:05:29.774684 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd0ea5f0-7993-48d3-b971-75e47aca2deb service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a7f7afc3d44ec788c05d005e52a1b8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:05:29.775313 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd0ea5f0-7993-48d3-b971-75e47aca2deb service swift] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:05:29.775596 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd0ea5f0-7993-48d3-b971-75e47aca2deb service swift] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:05:29.775864 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd0ea5f0-7993-48d3-b971-75e47aca2deb service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:05:29.777485 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-bd0ea5f0-7993-48d3-b971-75e47aca2deb service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:05:29.859791 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd0ea5f0-7993-48d3-b971-75e47aca2deb service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:05:29.861335 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd0ea5f0-7993-48d3-b971-75e47aca2deb service swift] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:05:29.869252 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 693/1400] 10.222.0.77 () {60 vars in 1384 bytes} [Tue Aug 30 05:05:29 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 342 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 30 05:05:29.998542 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-082710a2-95fa-4a15-aaf8-eba47da98495 None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:05:29.998847 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-082710a2-95fa-4a15-aaf8-eba47da98495 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:05:29.999135 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-082710a2-95fa-4a15-aaf8-eba47da98495 None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:05:30.000479 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 708/1401] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:05:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:05:30.009074 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96bce3ae-3d7e-462d-89e9-110e63bca6cb None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:05:30.009361 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96bce3ae-3d7e-462d-89e9-110e63bca6cb None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:05:30.009636 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96bce3ae-3d7e-462d-89e9-110e63bca6cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:05:30.083552 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-96bce3ae-3d7e-462d-89e9-110e63bca6cb None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:05:30.084789 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-96bce3ae-3d7e-462d-89e9-110e63bca6cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:05:30.148869 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:05:30.198212 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 694/1402] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:05:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:05:30.495867 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-082710a2-95fa-4a15-aaf8-eba47da98495 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:05:30.497414 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-082710a2-95fa-4a15-aaf8-eba47da98495 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:05:30.620575 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:05:30.668380 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-082710a2-95fa-4a15-aaf8-eba47da98495 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:05:30.800851 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cbbe41b-94f5-4260-933c-b4a1ac37254a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '886eb5e00bbf4ec292eb962b665b37d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:05:30.801492 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cbbe41b-94f5-4260-933c-b4a1ac37254a service nova] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:05:30.801792 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cbbe41b-94f5-4260-933c-b4a1ac37254a service nova] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:05:30.802064 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cbbe41b-94f5-4260-933c-b4a1ac37254a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:05:30.803708 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-4cbbe41b-94f5-4260-933c-b4a1ac37254a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:05:30.888523 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:05:30.913756 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cbbe41b-94f5-4260-933c-b4a1ac37254a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:05:30.915566 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cbbe41b-94f5-4260-933c-b4a1ac37254a service nova] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:05:30.943653 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 709/1403] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:05:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 447 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:05:32.920902 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21ffbe06-4c36-4c75-9888-35ee70d3ae7b None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:05:32.921512 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21ffbe06-4c36-4c75-9888-35ee70d3ae7b None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:05:32.921801 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21ffbe06-4c36-4c75-9888-35ee70d3ae7b None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:05:32.923232 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 695/1404] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:05:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:05:32.929046 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-310aece2-04b5-4d54-a58e-ed537932a3c2 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:05:32.929576 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-310aece2-04b5-4d54-a58e-ed537932a3c2 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:05:32.929669 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-310aece2-04b5-4d54-a58e-ed537932a3c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:05:33.003128 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-310aece2-04b5-4d54-a58e-ed537932a3c2 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:05:33.003128 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-310aece2-04b5-4d54-a58e-ed537932a3c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:05:33.109768 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 710/1405] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:05:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:05:33.403439 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21ffbe06-4c36-4c75-9888-35ee70d3ae7b None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:05:33.404983 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-21ffbe06-4c36-4c75-9888-35ee70d3ae7b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:05:33.576721 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21ffbe06-4c36-4c75-9888-35ee70d3ae7b None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:05:33.624072 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:05:33.705962 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa93ca4d-1a2d-4799-93fb-76d8cc90d9c9 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '886eb5e00bbf4ec292eb962b665b37d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:05:33.706608 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa93ca4d-1a2d-4799-93fb-76d8cc90d9c9 service nova] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:05:33.706983 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa93ca4d-1a2d-4799-93fb-76d8cc90d9c9 service nova] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:05:33.707385 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa93ca4d-1a2d-4799-93fb-76d8cc90d9c9 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:05:33.709056 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-aa93ca4d-1a2d-4799-93fb-76d8cc90d9c9 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:05:33.812570 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa93ca4d-1a2d-4799-93fb-76d8cc90d9c9 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:05:33.814184 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa93ca4d-1a2d-4799-93fb-76d8cc90d9c9 service nova] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:05:33.836937 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 696/1406] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:05:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 434 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:05:35.766410 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc561287-7b4b-4d99-9c88-84707a6d64db None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:05:35.767466 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc561287-7b4b-4d99-9c88-84707a6d64db None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:05:35.767466 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc561287-7b4b-4d99-9c88-84707a6d64db None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:05:35.768893 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 711/1407] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:05:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:05:35.773846 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fadbfb40-e40c-436f-8255-2d0d616c6eca None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:05:35.774292 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fadbfb40-e40c-436f-8255-2d0d616c6eca None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:05:35.774381 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fadbfb40-e40c-436f-8255-2d0d616c6eca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:05:35.827286 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:05:35.841865 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-fadbfb40-e40c-436f-8255-2d0d616c6eca None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:05:35.843314 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-fadbfb40-e40c-436f-8255-2d0d616c6eca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:05:35.959174 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 697/1408] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:05:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:05:36.286642 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc561287-7b4b-4d99-9c88-84707a6d64db None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:05:36.289953 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-bc561287-7b4b-4d99-9c88-84707a6d64db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:05:36.378676 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:05:36.494768 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc561287-7b4b-4d99-9c88-84707a6d64db None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:05:36.621398 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-528d962f-bb58-4c1c-bccd-7c61249b563b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '886eb5e00bbf4ec292eb962b665b37d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:05:36.622004 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-528d962f-bb58-4c1c-bccd-7c61249b563b service nova] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:05:36.622284 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-528d962f-bb58-4c1c-bccd-7c61249b563b service nova] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:05:36.622549 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-528d962f-bb58-4c1c-bccd-7c61249b563b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:05:36.624516 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-528d962f-bb58-4c1c-bccd-7c61249b563b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:05:36.634511 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:05:36.730559 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-528d962f-bb58-4c1c-bccd-7c61249b563b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:05:36.732455 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-528d962f-bb58-4c1c-bccd-7c61249b563b service nova] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:05:36.753527 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 712/1409] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:05:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 468 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:05:45.957080 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=46 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:05:46.748629 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=41 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:05:51.748485 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-476a52f5-dbcd-4a57-9e1b-03be8d77ce8f None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:05:51.749022 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-476a52f5-dbcd-4a57-9e1b-03be8d77ce8f None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:05:51.749022 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-476a52f5-dbcd-4a57-9e1b-03be8d77ce8f None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:05:51.750394 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 698/1410] 10.222.0.77 () {54 vars in 918 bytes} [Tue Aug 30 05:05:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:05:51.755646 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2447c431-1548-4bdd-b4b2-fd4086336cad None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:05:51.756066 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2447c431-1548-4bdd-b4b2-fd4086336cad None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:05:51.756175 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2447c431-1548-4bdd-b4b2-fd4086336cad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:05:51.830976 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-2447c431-1548-4bdd-b4b2-fd4086336cad None None] MFA Rules not processed for user `1972959402704b89925001cd8d90cb0e`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:05:51.832172 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-2447c431-1548-4bdd-b4b2-fd4086336cad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:05:51.937528 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 713/1411] 10.222.0.77 () {58 vars in 1015 bytes} [Tue Aug 30 05:05:51 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:05:51.951363 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c9ea832-4553-40b9-b1cb-65a34d5234f1 None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:05:51.951649 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c9ea832-4553-40b9-b1cb-65a34d5234f1 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:05:51.951925 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c9ea832-4553-40b9-b1cb-65a34d5234f1 None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:05:51.953260 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 699/1412] 10.222.0.77 () {54 vars in 918 bytes} [Tue Aug 30 05:05:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:05:51.958360 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d8a277d-4483-4bd3-a1ce-216a50586d75 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:05:51.958642 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d8a277d-4483-4bd3-a1ce-216a50586d75 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:05:51.958947 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d8a277d-4483-4bd3-a1ce-216a50586d75 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:05:52.030239 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:05:52.033816 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-9d8a277d-4483-4bd3-a1ce-216a50586d75 None None] MFA Rules not processed for user `886eb5e00bbf4ec292eb962b665b37d0`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:05:52.035659 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-9d8a277d-4483-4bd3-a1ce-216a50586d75 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:05:52.148829 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 714/1413] 10.222.0.77 () {58 vars in 1015 bytes} [Tue Aug 30 05:05:51 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:05:54.515622 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e8a9e3b-e9c8-48a3-9ec7-cd301063bc95 None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:05:54.516722 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e8a9e3b-e9c8-48a3-9ec7-cd301063bc95 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:05:54.516722 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e8a9e3b-e9c8-48a3-9ec7-cd301063bc95 None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:05:54.518083 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 700/1414] 127.0.0.1 () {54 vars in 917 bytes} [Tue Aug 30 05:05:54 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:05:54.523278 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d25f3ca2-e3df-45e6-bc5b-1d89b939cec0 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:05:54.523740 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d25f3ca2-e3df-45e6-bc5b-1d89b939cec0 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:05:54.523839 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d25f3ca2-e3df-45e6-bc5b-1d89b939cec0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:05:54.589383 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-d25f3ca2-e3df-45e6-bc5b-1d89b939cec0 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:05:54.590595 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-d25f3ca2-e3df-45e6-bc5b-1d89b939cec0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:05:54.645570 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:05:54.686909 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 715/1415] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Aug 30 05:05:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:05:54.973685 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e8a9e3b-e9c8-48a3-9ec7-cd301063bc95 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:05:54.975542 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-3e8a9e3b-e9c8-48a3-9ec7-cd301063bc95 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:05:55.119424 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e8a9e3b-e9c8-48a3-9ec7-cd301063bc95 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:05:55.195569 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:05:55.236701 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6015750-01b0-4aeb-afe5-5f68498ccc9f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '886eb5e00bbf4ec292eb962b665b37d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:05:55.237235 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6015750-01b0-4aeb-afe5-5f68498ccc9f service nova] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:05:55.237518 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6015750-01b0-4aeb-afe5-5f68498ccc9f service nova] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:05:55.237791 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6015750-01b0-4aeb-afe5-5f68498ccc9f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:05:55.239467 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-f6015750-01b0-4aeb-afe5-5f68498ccc9f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:05:55.327923 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6015750-01b0-4aeb-afe5-5f68498ccc9f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:05:55.329510 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6015750-01b0-4aeb-afe5-5f68498ccc9f service nova] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:05:55.347304 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 701/1416] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:05:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 374 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:05:56.233841 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6513dde-4cae-45ea-87e7-cadc54d9f0b2 None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:05:56.234410 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6513dde-4cae-45ea-87e7-cadc54d9f0b2 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:05:56.234410 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6513dde-4cae-45ea-87e7-cadc54d9f0b2 None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:05:56.235791 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 716/1417] 127.0.0.1 () {54 vars in 901 bytes} [Tue Aug 30 05:05:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:05:56.240821 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da6aa13e-0052-4826-8904-2a108cbc7d19 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:05:56.241235 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da6aa13e-0052-4826-8904-2a108cbc7d19 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:05:56.241370 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da6aa13e-0052-4826-8904-2a108cbc7d19 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:05:56.316571 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-da6aa13e-0052-4826-8904-2a108cbc7d19 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:05:56.317772 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-da6aa13e-0052-4826-8904-2a108cbc7d19 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:05:56.343771 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:05:56.412926 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 702/1418] 127.0.0.1 () {58 vars in 998 bytes} [Tue Aug 30 05:05:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:05:56.564475 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6513dde-4cae-45ea-87e7-cadc54d9f0b2 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:05:56.566017 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-e6513dde-4cae-45ea-87e7-cadc54d9f0b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:05:56.711436 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6513dde-4cae-45ea-87e7-cadc54d9f0b2 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:05:56.751910 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:05:56.830783 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff0c3d29-34da-4e94-a410-10401855ac87 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '886eb5e00bbf4ec292eb962b665b37d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:05:56.831324 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff0c3d29-34da-4e94-a410-10401855ac87 service nova] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:05:56.831603 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff0c3d29-34da-4e94-a410-10401855ac87 service nova] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:05:56.831863 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff0c3d29-34da-4e94-a410-10401855ac87 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:05:56.833432 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-ff0c3d29-34da-4e94-a410-10401855ac87 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:05:56.920343 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff0c3d29-34da-4e94-a410-10401855ac87 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:05:56.921927 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff0c3d29-34da-4e94-a410-10401855ac87 service nova] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:05:56.940489 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 717/1419] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:05:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 376 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:05:57.818390 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-083f505b-604b-4c1e-8144-2fb6d64878a8 None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:05:57.818976 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-083f505b-604b-4c1e-8144-2fb6d64878a8 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:05:57.818976 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-083f505b-604b-4c1e-8144-2fb6d64878a8 None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:05:57.820345 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 703/1420] 127.0.0.1 () {54 vars in 901 bytes} [Tue Aug 30 05:05:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:05:57.825290 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6b5ee07-caac-4745-b505-145dec98e4d0 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:05:57.825724 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6b5ee07-caac-4745-b505-145dec98e4d0 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:05:57.825821 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6b5ee07-caac-4745-b505-145dec98e4d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:05:57.880840 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:05:57.902417 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-c6b5ee07-caac-4745-b505-145dec98e4d0 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:05:57.903646 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-c6b5ee07-caac-4745-b505-145dec98e4d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:05:57.997887 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 718/1421] 127.0.0.1 () {58 vars in 998 bytes} [Tue Aug 30 05:05:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:05:58.150142 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-083f505b-604b-4c1e-8144-2fb6d64878a8 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:05:58.151647 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-083f505b-604b-4c1e-8144-2fb6d64878a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:05:58.307347 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-083f505b-604b-4c1e-8144-2fb6d64878a8 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:05:58.319189 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:05:58.419883 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eecdf55d-cf32-4d2e-b5b9-098433a88da3 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '886eb5e00bbf4ec292eb962b665b37d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:05:58.420402 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eecdf55d-cf32-4d2e-b5b9-098433a88da3 service nova] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:05:58.420683 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eecdf55d-cf32-4d2e-b5b9-098433a88da3 service nova] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:05:58.420952 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eecdf55d-cf32-4d2e-b5b9-098433a88da3 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:05:58.423250 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-eecdf55d-cf32-4d2e-b5b9-098433a88da3 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:05:58.511874 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eecdf55d-cf32-4d2e-b5b9-098433a88da3 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:05:58.513476 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eecdf55d-cf32-4d2e-b5b9-098433a88da3 service nova] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:05:58.532073 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 704/1422] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:05:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 382 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:05:59.379377 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeea6671-455a-4f68-9f1e-da397e7a80ed None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:05:59.379948 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeea6671-455a-4f68-9f1e-da397e7a80ed None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:05:59.379948 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeea6671-455a-4f68-9f1e-da397e7a80ed None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:05:59.381285 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 719/1423] 127.0.0.1 () {54 vars in 901 bytes} [Tue Aug 30 05:05:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:05:59.386389 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c1d175e-b36e-474d-9cff-d73e90552f4f None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:05:59.386789 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c1d175e-b36e-474d-9cff-d73e90552f4f None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:05:59.386981 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c1d175e-b36e-474d-9cff-d73e90552f4f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:05:59.406273 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:05:59.464549 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-0c1d175e-b36e-474d-9cff-d73e90552f4f None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:05:59.465777 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-0c1d175e-b36e-474d-9cff-d73e90552f4f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:05:59.560747 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 705/1424] 127.0.0.1 () {58 vars in 998 bytes} [Tue Aug 30 05:05:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:05:59.712266 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeea6671-455a-4f68-9f1e-da397e7a80ed None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:05:59.713737 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-eeea6671-455a-4f68-9f1e-da397e7a80ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:05:59.823186 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:05:59.859080 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeea6671-455a-4f68-9f1e-da397e7a80ed None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:05:59.971051 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfcfbb3b-45db-4c93-8043-2dd71db46dd3 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '886eb5e00bbf4ec292eb962b665b37d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:05:59.971572 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfcfbb3b-45db-4c93-8043-2dd71db46dd3 service nova] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:05:59.971849 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfcfbb3b-45db-4c93-8043-2dd71db46dd3 service nova] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:05:59.972118 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfcfbb3b-45db-4c93-8043-2dd71db46dd3 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:05:59.973689 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-cfcfbb3b-45db-4c93-8043-2dd71db46dd3 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:06:00.048549 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:06:00.061890 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfcfbb3b-45db-4c93-8043-2dd71db46dd3 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:06:00.063523 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfcfbb3b-45db-4c93-8043-2dd71db46dd3 service nova] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:06:00.081562 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 720/1425] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:05:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 370 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:06:00.930331 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5594a5ec-d8f4-4b43-aa21-9b686e75943d None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:06:00.930968 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5594a5ec-d8f4-4b43-aa21-9b686e75943d None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:06:00.930968 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5594a5ec-d8f4-4b43-aa21-9b686e75943d None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:06:00.932637 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 706/1426] 127.0.0.1 () {54 vars in 901 bytes} [Tue Aug 30 05:06:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:06:00.937481 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8478f823-5937-453b-9997-fc442e6fa3be None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:06:00.938488 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8478f823-5937-453b-9997-fc442e6fa3be None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:06:00.938612 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8478f823-5937-453b-9997-fc442e6fa3be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:06:01.012956 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-8478f823-5937-453b-9997-fc442e6fa3be None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:06:01.014144 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-8478f823-5937-453b-9997-fc442e6fa3be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:06:01.107879 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 721/1427] 127.0.0.1 () {58 vars in 998 bytes} [Tue Aug 30 05:06:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:06:01.257115 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5594a5ec-d8f4-4b43-aa21-9b686e75943d None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:06:01.258603 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-5594a5ec-d8f4-4b43-aa21-9b686e75943d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:06:01.340741 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:06:01.401793 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5594a5ec-d8f4-4b43-aa21-9b686e75943d None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:06:01.513654 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f5481fc-0204-4324-b7ca-6532a706b95c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '886eb5e00bbf4ec292eb962b665b37d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:06:01.514198 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f5481fc-0204-4324-b7ca-6532a706b95c service nova] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:06:01.514480 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f5481fc-0204-4324-b7ca-6532a706b95c service nova] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:06:01.514752 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f5481fc-0204-4324-b7ca-6532a706b95c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:06:01.516909 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-8f5481fc-0204-4324-b7ca-6532a706b95c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:06:01.563906 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:06:01.606716 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f5481fc-0204-4324-b7ca-6532a706b95c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:06:01.608349 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f5481fc-0204-4324-b7ca-6532a706b95c service nova] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:06:01.625858 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 707/1428] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:06:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 369 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:06:02.425040 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0154dc7e-ba71-4c49-96ee-ee4bb519f8bb None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:06:02.425697 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0154dc7e-ba71-4c49-96ee-ee4bb519f8bb None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:06:02.425697 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0154dc7e-ba71-4c49-96ee-ee4bb519f8bb None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:06:02.427003 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 722/1429] 127.0.0.1 () {54 vars in 901 bytes} [Tue Aug 30 05:06:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:06:02.431843 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fa32c48-317a-4fa4-9f0b-42a334c5854b None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:06:02.432303 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fa32c48-317a-4fa4-9f0b-42a334c5854b None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:06:02.432409 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fa32c48-317a-4fa4-9f0b-42a334c5854b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:06:02.508761 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-0fa32c48-317a-4fa4-9f0b-42a334c5854b None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:06:02.509964 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-0fa32c48-317a-4fa4-9f0b-42a334c5854b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:06:02.602984 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 708/1430] 127.0.0.1 () {58 vars in 998 bytes} [Tue Aug 30 05:06:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:06:02.753807 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0154dc7e-ba71-4c49-96ee-ee4bb519f8bb None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:06:02.755304 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-0154dc7e-ba71-4c49-96ee-ee4bb519f8bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:06:02.803130 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:06:02.899278 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0154dc7e-ba71-4c49-96ee-ee4bb519f8bb None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:06:03.012958 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa7b5d78-708a-43f1-a87b-621b92de9364 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '886eb5e00bbf4ec292eb962b665b37d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:06:03.013481 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa7b5d78-708a-43f1-a87b-621b92de9364 service nova] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:06:03.013756 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa7b5d78-708a-43f1-a87b-621b92de9364 service nova] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:06:03.014022 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa7b5d78-708a-43f1-a87b-621b92de9364 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:06:03.016114 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-fa7b5d78-708a-43f1-a87b-621b92de9364 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:06:03.022929 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:06:03.104279 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa7b5d78-708a-43f1-a87b-621b92de9364 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:06:03.105873 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa7b5d78-708a-43f1-a87b-621b92de9364 service nova] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:06:03.123484 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 723/1431] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:06:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 370 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:06:03.925398 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e93d82-7d2c-4c87-bd18-255a5d33629d None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:06:03.925965 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e93d82-7d2c-4c87-bd18-255a5d33629d None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:06:03.925965 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e93d82-7d2c-4c87-bd18-255a5d33629d None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:06:03.927316 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 709/1432] 127.0.0.1 () {54 vars in 901 bytes} [Tue Aug 30 05:06:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:06:03.932453 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-069b6a19-b501-4808-a435-3b79c2e845d9 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:06:03.932874 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-069b6a19-b501-4808-a435-3b79c2e845d9 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:06:03.933014 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-069b6a19-b501-4808-a435-3b79c2e845d9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:06:04.009240 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-069b6a19-b501-4808-a435-3b79c2e845d9 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:06:04.010488 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-069b6a19-b501-4808-a435-3b79c2e845d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:06:04.077332 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:06:04.106128 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 724/1433] 127.0.0.1 () {58 vars in 998 bytes} [Tue Aug 30 05:06:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:06:04.259453 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96e93d82-7d2c-4c87-bd18-255a5d33629d None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:06:04.260900 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-96e93d82-7d2c-4c87-bd18-255a5d33629d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:06:04.278272 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:06:04.404916 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96e93d82-7d2c-4c87-bd18-255a5d33629d None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:06:04.491894 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:06:04.518270 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f657e2b-dea5-4b1b-bc34-6360edaf2588 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '886eb5e00bbf4ec292eb962b665b37d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:06:04.518785 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f657e2b-dea5-4b1b-bc34-6360edaf2588 service nova] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:06:04.519094 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f657e2b-dea5-4b1b-bc34-6360edaf2588 service nova] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:06:04.519365 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f657e2b-dea5-4b1b-bc34-6360edaf2588 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:06:04.520950 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-9f657e2b-dea5-4b1b-bc34-6360edaf2588 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:06:04.608988 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f657e2b-dea5-4b1b-bc34-6360edaf2588 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:06:04.610576 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f657e2b-dea5-4b1b-bc34-6360edaf2588 service nova] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:06:04.627988 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 710/1434] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:06:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 369 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:06:05.425600 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67e1cb50-b979-4723-bcb2-401c30bc6182 None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:06:05.426164 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67e1cb50-b979-4723-bcb2-401c30bc6182 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:06:05.426164 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67e1cb50-b979-4723-bcb2-401c30bc6182 None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:06:05.427500 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 725/1435] 127.0.0.1 () {54 vars in 901 bytes} [Tue Aug 30 05:06:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:06:05.432257 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dc42028-d34c-4c3c-a437-c95cc64e7f02 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:06:05.432672 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dc42028-d34c-4c3c-a437-c95cc64e7f02 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:06:05.432810 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dc42028-d34c-4c3c-a437-c95cc64e7f02 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:06:05.509335 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-6dc42028-d34c-4c3c-a437-c95cc64e7f02 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:06:05.510528 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-6dc42028-d34c-4c3c-a437-c95cc64e7f02 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:06:05.551235 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:06:05.605752 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 711/1436] 127.0.0.1 () {58 vars in 998 bytes} [Tue Aug 30 05:06:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:06:05.761333 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67e1cb50-b979-4723-bcb2-401c30bc6182 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:06:05.762744 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-67e1cb50-b979-4723-bcb2-401c30bc6182 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:06:05.907521 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67e1cb50-b979-4723-bcb2-401c30bc6182 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:06:05.962323 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:06:06.021259 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-240a5c8a-6ae4-4aa6-8110-ed98f84f9127 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '886eb5e00bbf4ec292eb962b665b37d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:06:06.021799 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-240a5c8a-6ae4-4aa6-8110-ed98f84f9127 service nova] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:06:06.022078 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-240a5c8a-6ae4-4aa6-8110-ed98f84f9127 service nova] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:06:06.022353 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-240a5c8a-6ae4-4aa6-8110-ed98f84f9127 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:06:06.023986 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-240a5c8a-6ae4-4aa6-8110-ed98f84f9127 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:06:06.111672 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-240a5c8a-6ae4-4aa6-8110-ed98f84f9127 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:06:06.113257 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-240a5c8a-6ae4-4aa6-8110-ed98f84f9127 service nova] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:06:06.130707 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 726/1437] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:06:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 370 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:06:06.935427 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e72b2ab-e6d0-4c7d-993e-9066026fc7bd None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:06:06.935992 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e72b2ab-e6d0-4c7d-993e-9066026fc7bd None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:06:06.935992 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e72b2ab-e6d0-4c7d-993e-9066026fc7bd None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:06:06.937357 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 712/1438] 127.0.0.1 () {54 vars in 901 bytes} [Tue Aug 30 05:06:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:06:06.942871 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72b2a41d-cc77-4dde-8f52-f7a5e8a8d0b7 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:06:06.943287 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72b2a41d-cc77-4dde-8f52-f7a5e8a8d0b7 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:06:06.943427 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72b2a41d-cc77-4dde-8f52-f7a5e8a8d0b7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:06:07.019122 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-72b2a41d-cc77-4dde-8f52-f7a5e8a8d0b7 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:06:07.020259 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-72b2a41d-cc77-4dde-8f52-f7a5e8a8d0b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:06:07.022265 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:06:07.116049 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 727/1439] 127.0.0.1 () {58 vars in 998 bytes} [Tue Aug 30 05:06:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:06:07.269256 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e72b2ab-e6d0-4c7d-993e-9066026fc7bd None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:06:07.270694 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-1e72b2ab-e6d0-4c7d-993e-9066026fc7bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:06:07.412664 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e72b2ab-e6d0-4c7d-993e-9066026fc7bd None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:06:07.439168 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:06:07.525559 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b2d489f-2e9d-4080-bf56-321850cdcb79 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '886eb5e00bbf4ec292eb962b665b37d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:06:07.526086 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b2d489f-2e9d-4080-bf56-321850cdcb79 service nova] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:06:07.526361 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b2d489f-2e9d-4080-bf56-321850cdcb79 service nova] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:06:07.526628 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b2d489f-2e9d-4080-bf56-321850cdcb79 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:06:07.528225 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-0b2d489f-2e9d-4080-bf56-321850cdcb79 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:06:07.617180 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b2d489f-2e9d-4080-bf56-321850cdcb79 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:06:07.618791 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b2d489f-2e9d-4080-bf56-321850cdcb79 service nova] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:06:07.636729 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 713/1440] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:06:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 368 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:06:09.455866 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4e343cb-1efa-40ce-93a8-740eb6d92872 None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:06:09.456402 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4e343cb-1efa-40ce-93a8-740eb6d92872 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:06:09.456402 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4e343cb-1efa-40ce-93a8-740eb6d92872 None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:06:09.457815 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 728/1441] 127.0.0.1 () {54 vars in 917 bytes} [Tue Aug 30 05:06:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:06:09.463291 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e669d1a-59ef-4952-8d42-97942f497354 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:06:09.463760 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e669d1a-59ef-4952-8d42-97942f497354 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:06:09.463968 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e669d1a-59ef-4952-8d42-97942f497354 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:06:09.504733 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:06:09.530014 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-6e669d1a-59ef-4952-8d42-97942f497354 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:06:09.531236 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-6e669d1a-59ef-4952-8d42-97942f497354 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:06:09.624366 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 714/1442] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Aug 30 05:06:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:06:09.907517 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4e343cb-1efa-40ce-93a8-740eb6d92872 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:06:09.909098 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-d4e343cb-1efa-40ce-93a8-740eb6d92872 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:06:10.016642 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4e343cb-1efa-40ce-93a8-740eb6d92872 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:06:10.048688 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:06:10.090109 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-055ea951-cf31-4d7b-bdcb-9c13292750ec service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:06:10.090703 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-055ea951-cf31-4d7b-bdcb-9c13292750ec service neutron] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:06:10.091009 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-055ea951-cf31-4d7b-bdcb-9c13292750ec service neutron] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:06:10.091278 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-055ea951-cf31-4d7b-bdcb-9c13292750ec service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:06:10.092858 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-055ea951-cf31-4d7b-bdcb-9c13292750ec service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:06:10.200102 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-055ea951-cf31-4d7b-bdcb-9c13292750ec service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:06:10.201524 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-055ea951-cf31-4d7b-bdcb-9c13292750ec service neutron] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:06:10.220119 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 729/1443] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:06:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:06:12.252016 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d51c48-8c0f-422f-83cb-b9fceae36e81 None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:06:12.252598 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d51c48-8c0f-422f-83cb-b9fceae36e81 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:06:12.252598 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d51c48-8c0f-422f-83cb-b9fceae36e81 None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:06:12.253955 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 715/1444] 127.0.0.1 () {54 vars in 917 bytes} [Tue Aug 30 05:06:12 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:06:12.259239 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c49b821-2ccb-43a3-b388-98b9d80d02af None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:06:12.259696 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c49b821-2ccb-43a3-b388-98b9d80d02af None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:06:12.259803 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c49b821-2ccb-43a3-b388-98b9d80d02af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:06:12.326291 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-4c49b821-2ccb-43a3-b388-98b9d80d02af None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:06:12.327513 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-4c49b821-2ccb-43a3-b388-98b9d80d02af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:06:12.339988 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:06:12.422868 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 730/1445] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Aug 30 05:06:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 05:06:12.708070 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8d51c48-8c0f-422f-83cb-b9fceae36e81 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:06:12.709596 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-a8d51c48-8c0f-422f-83cb-b9fceae36e81 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:06:12.810704 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8d51c48-8c0f-422f-83cb-b9fceae36e81 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:06:12.822930 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:06:12.882947 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc598a70-03a3-420d-80ee-c30b17f3cd30 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:06:12.883541 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc598a70-03a3-420d-80ee-c30b17f3cd30 service neutron] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:06:12.883819 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc598a70-03a3-420d-80ee-c30b17f3cd30 service neutron] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:06:12.884092 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc598a70-03a3-420d-80ee-c30b17f3cd30 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:06:12.885656 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-dc598a70-03a3-420d-80ee-c30b17f3cd30 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:06:12.992028 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc598a70-03a3-420d-80ee-c30b17f3cd30 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:06:12.993371 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc598a70-03a3-420d-80ee-c30b17f3cd30 service neutron] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:06:13.011336 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 716/1446] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:06:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 304 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 05:06:15.196331 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f188600d-4b02-4c83-bf45-a5e0ebfa843f None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:06:15.196858 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f188600d-4b02-4c83-bf45-a5e0ebfa843f None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:06:15.196858 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f188600d-4b02-4c83-bf45-a5e0ebfa843f None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:06:15.198246 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 731/1447] 127.0.0.1 () {54 vars in 917 bytes} [Tue Aug 30 05:06:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:06:15.203399 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fcd6bc7-b17d-4549-a44b-2a5ea6ee9dcd None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:06:15.203804 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fcd6bc7-b17d-4549-a44b-2a5ea6ee9dcd None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:06:15.203932 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fcd6bc7-b17d-4549-a44b-2a5ea6ee9dcd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:06:15.253283 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:06:15.270989 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-1fcd6bc7-b17d-4549-a44b-2a5ea6ee9dcd None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:06:15.272204 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-1fcd6bc7-b17d-4549-a44b-2a5ea6ee9dcd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:06:15.366915 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 717/1448] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Aug 30 05:06:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:06:15.649149 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f188600d-4b02-4c83-bf45-a5e0ebfa843f None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:06:15.650702 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-f188600d-4b02-4c83-bf45-a5e0ebfa843f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:06:15.752671 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f188600d-4b02-4c83-bf45-a5e0ebfa843f None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:06:15.793932 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:06:15.827282 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2273f2c-220b-48e2-a9fe-eecc920e8e37 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:06:15.828381 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2273f2c-220b-48e2-a9fe-eecc920e8e37 service neutron] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:06:15.828661 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2273f2c-220b-48e2-a9fe-eecc920e8e37 service neutron] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:06:15.828938 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2273f2c-220b-48e2-a9fe-eecc920e8e37 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:06:15.830519 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-d2273f2c-220b-48e2-a9fe-eecc920e8e37 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:06:15.938853 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2273f2c-220b-48e2-a9fe-eecc920e8e37 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:06:15.940595 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2273f2c-220b-48e2-a9fe-eecc920e8e37 service neutron] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:06:15.958836 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 732/1449] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:06:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:06:17.958572 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d902acc7-a84b-4d61-a7ab-d9bab3cda449 None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:06:17.959198 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d902acc7-a84b-4d61-a7ab-d9bab3cda449 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:06:17.959198 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d902acc7-a84b-4d61-a7ab-d9bab3cda449 None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:06:17.960492 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 718/1450] 127.0.0.1 () {54 vars in 917 bytes} [Tue Aug 30 05:06:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:06:17.965758 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61df71bb-ab0e-471e-a995-36b8ba5ded1f None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:06:17.966189 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61df71bb-ab0e-471e-a995-36b8ba5ded1f None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:06:17.966302 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61df71bb-ab0e-471e-a995-36b8ba5ded1f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:06:18.034906 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-61df71bb-ab0e-471e-a995-36b8ba5ded1f None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:06:18.036101 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-61df71bb-ab0e-471e-a995-36b8ba5ded1f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:06:18.056105 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:06:18.131477 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 733/1451] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Aug 30 05:06:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:06:18.412877 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d902acc7-a84b-4d61-a7ab-d9bab3cda449 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:06:18.414428 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-d902acc7-a84b-4d61-a7ab-d9bab3cda449 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:06:18.517902 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d902acc7-a84b-4d61-a7ab-d9bab3cda449 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:06:18.537095 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:06:18.593087 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-228bdbda-4ef4-44e3-b0c4-110067e3cfd0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:06:18.593680 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-228bdbda-4ef4-44e3-b0c4-110067e3cfd0 service neutron] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:06:18.593992 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-228bdbda-4ef4-44e3-b0c4-110067e3cfd0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:06:18.594256 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-228bdbda-4ef4-44e3-b0c4-110067e3cfd0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:06:18.595859 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-228bdbda-4ef4-44e3-b0c4-110067e3cfd0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:06:18.701950 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-228bdbda-4ef4-44e3-b0c4-110067e3cfd0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:06:18.703891 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-228bdbda-4ef4-44e3-b0c4-110067e3cfd0 service neutron] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:06:18.721263 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 719/1452] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:06:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:06:20.679826 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-065f9ed7-601e-47c4-9add-29611c58cdc9 None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:06:20.680376 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-065f9ed7-601e-47c4-9add-29611c58cdc9 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:06:20.680376 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-065f9ed7-601e-47c4-9add-29611c58cdc9 None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:06:20.681770 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 734/1453] 127.0.0.1 () {54 vars in 917 bytes} [Tue Aug 30 05:06:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:06:20.687457 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-765dc0e4-a226-4ad2-af9b-105ec50476ac None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:06:20.687855 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-765dc0e4-a226-4ad2-af9b-105ec50476ac None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:06:20.688021 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-765dc0e4-a226-4ad2-af9b-105ec50476ac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:06:20.751120 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:06:20.753860 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-765dc0e4-a226-4ad2-af9b-105ec50476ac None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:06:20.755633 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-765dc0e4-a226-4ad2-af9b-105ec50476ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:06:20.853705 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 720/1454] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Aug 30 05:06:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:06:21.138345 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-065f9ed7-601e-47c4-9add-29611c58cdc9 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:06:21.139919 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-065f9ed7-601e-47c4-9add-29611c58cdc9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:06:21.241058 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-065f9ed7-601e-47c4-9add-29611c58cdc9 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:06:21.288585 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:06:21.314268 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c28bfee-358d-49b8-b910-2a7c8ae93974 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:06:21.314884 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c28bfee-358d-49b8-b910-2a7c8ae93974 service neutron] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:06:21.315163 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c28bfee-358d-49b8-b910-2a7c8ae93974 service neutron] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:06:21.315429 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c28bfee-358d-49b8-b910-2a7c8ae93974 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:06:21.317029 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-8c28bfee-358d-49b8-b910-2a7c8ae93974 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:06:21.425539 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c28bfee-358d-49b8-b910-2a7c8ae93974 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:06:21.426909 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c28bfee-358d-49b8-b910-2a7c8ae93974 service neutron] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:06:21.444314 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 735/1455] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:06:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 306 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:06:23.381822 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-614edb61-4015-41ae-a380-7cff99eaa467 None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:06:23.382420 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-614edb61-4015-41ae-a380-7cff99eaa467 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:06:23.382420 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-614edb61-4015-41ae-a380-7cff99eaa467 None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:06:23.383785 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 721/1456] 127.0.0.1 () {54 vars in 917 bytes} [Tue Aug 30 05:06:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:06:23.388898 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aba484ab-0136-4099-ba59-13e6512b6d82 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:06:23.389299 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aba484ab-0136-4099-ba59-13e6512b6d82 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:06:23.389445 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aba484ab-0136-4099-ba59-13e6512b6d82 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:06:23.453174 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-aba484ab-0136-4099-ba59-13e6512b6d82 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:06:23.454395 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-aba484ab-0136-4099-ba59-13e6512b6d82 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:06:23.482948 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:06:23.550283 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 736/1457] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Aug 30 05:06:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:06:23.835653 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-614edb61-4015-41ae-a380-7cff99eaa467 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:06:23.837199 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-614edb61-4015-41ae-a380-7cff99eaa467 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:06:23.939654 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-614edb61-4015-41ae-a380-7cff99eaa467 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:06:23.966887 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:06:24.011713 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4826e378-194c-4278-87a3-93b7260b6913 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:06:24.012342 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4826e378-194c-4278-87a3-93b7260b6913 service neutron] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:06:24.012627 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4826e378-194c-4278-87a3-93b7260b6913 service neutron] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:06:24.012904 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4826e378-194c-4278-87a3-93b7260b6913 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:06:24.014516 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-4826e378-194c-4278-87a3-93b7260b6913 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:06:24.122276 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4826e378-194c-4278-87a3-93b7260b6913 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:06:24.123726 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4826e378-194c-4278-87a3-93b7260b6913 service neutron] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:06:24.141602 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 722/1458] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:06:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:06:26.167595 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c15b385d-467b-40a4-b9e0-70d421c364ed None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:06:26.168157 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c15b385d-467b-40a4-b9e0-70d421c364ed None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:06:26.168157 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c15b385d-467b-40a4-b9e0-70d421c364ed None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:06:26.169458 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 737/1459] 127.0.0.1 () {54 vars in 917 bytes} [Tue Aug 30 05:06:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:06:26.175209 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ab90a19-fd7b-4211-a475-3482364c9742 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:06:26.175624 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ab90a19-fd7b-4211-a475-3482364c9742 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:06:26.175744 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ab90a19-fd7b-4211-a475-3482364c9742 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:06:26.240620 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-7ab90a19-fd7b-4211-a475-3482364c9742 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:06:26.241824 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-7ab90a19-fd7b-4211-a475-3482364c9742 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:06:26.249976 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:06:26.336583 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 723/1460] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Aug 30 05:06:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:06:26.618462 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c15b385d-467b-40a4-b9e0-70d421c364ed None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:06:26.620050 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-c15b385d-467b-40a4-b9e0-70d421c364ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:06:26.722616 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c15b385d-467b-40a4-b9e0-70d421c364ed None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:06:26.777239 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:06:26.795111 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c7585d7-0a53-4b5f-b032-af0af9eec54d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:06:26.795704 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c7585d7-0a53-4b5f-b032-af0af9eec54d service neutron] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:06:26.795979 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c7585d7-0a53-4b5f-b032-af0af9eec54d service neutron] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:06:26.796244 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c7585d7-0a53-4b5f-b032-af0af9eec54d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:06:26.797791 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-0c7585d7-0a53-4b5f-b032-af0af9eec54d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:06:26.903959 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c7585d7-0a53-4b5f-b032-af0af9eec54d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:06:26.905372 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c7585d7-0a53-4b5f-b032-af0af9eec54d service neutron] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:06:26.922909 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 738/1461] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:06:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 305 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:06:28.864234 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2cdb419-aa1d-437a-a5b5-109d61238a7a None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:06:28.864797 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2cdb419-aa1d-437a-a5b5-109d61238a7a None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:06:28.864797 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2cdb419-aa1d-437a-a5b5-109d61238a7a None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:06:28.866117 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 724/1462] 127.0.0.1 () {54 vars in 917 bytes} [Tue Aug 30 05:06:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:06:28.871664 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1dad025-f69c-40ad-92a9-d060713a377f None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:06:28.872082 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1dad025-f69c-40ad-92a9-d060713a377f None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:06:28.872185 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1dad025-f69c-40ad-92a9-d060713a377f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:06:28.939697 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-c1dad025-f69c-40ad-92a9-d060713a377f None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:06:28.940901 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-c1dad025-f69c-40ad-92a9-d060713a377f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:06:28.978981 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:06:29.038546 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 739/1463] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Aug 30 05:06:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:06:29.324993 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2cdb419-aa1d-437a-a5b5-109d61238a7a None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:06:29.326533 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-c2cdb419-aa1d-437a-a5b5-109d61238a7a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:06:29.431949 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2cdb419-aa1d-437a-a5b5-109d61238a7a None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:06:29.470709 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:06:29.506983 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eea0d2b-a3f6-4305-ba9d-f014138866e5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:06:29.507577 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eea0d2b-a3f6-4305-ba9d-f014138866e5 service neutron] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:06:29.507849 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eea0d2b-a3f6-4305-ba9d-f014138866e5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:06:29.508112 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eea0d2b-a3f6-4305-ba9d-f014138866e5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:06:29.509670 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-6eea0d2b-a3f6-4305-ba9d-f014138866e5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:06:29.615628 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eea0d2b-a3f6-4305-ba9d-f014138866e5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:06:29.617078 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eea0d2b-a3f6-4305-ba9d-f014138866e5 service neutron] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:06:29.635397 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 725/1464] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:06:29 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:06:31.539914 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bde4189-4295-4265-ad5c-3325facc1897 None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:06:31.540412 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bde4189-4295-4265-ad5c-3325facc1897 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:06:31.540531 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bde4189-4295-4265-ad5c-3325facc1897 None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:06:31.541770 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 740/1465] 127.0.0.1 () {54 vars in 917 bytes} [Tue Aug 30 05:06:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:06:31.546955 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31fb5aa4-fbb5-484c-a05f-461af6077a46 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:06:31.547400 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31fb5aa4-fbb5-484c-a05f-461af6077a46 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:06:31.547513 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31fb5aa4-fbb5-484c-a05f-461af6077a46 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:06:31.613119 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-31fb5aa4-fbb5-484c-a05f-461af6077a46 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:06:31.613738 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-31fb5aa4-fbb5-484c-a05f-461af6077a46 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:06:31.631717 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:06:31.709956 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 726/1466] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Aug 30 05:06:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:06:31.994592 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bde4189-4295-4265-ad5c-3325facc1897 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:06:31.996158 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-8bde4189-4295-4265-ad5c-3325facc1897 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:06:32.097076 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bde4189-4295-4265-ad5c-3325facc1897 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:06:32.162870 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:06:32.170072 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a9df8bd-ad4f-46cb-be6d-bfa4e0e20e9c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:06:32.170668 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a9df8bd-ad4f-46cb-be6d-bfa4e0e20e9c service neutron] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:06:32.170979 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a9df8bd-ad4f-46cb-be6d-bfa4e0e20e9c service neutron] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:06:32.171251 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a9df8bd-ad4f-46cb-be6d-bfa4e0e20e9c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:06:32.172813 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-8a9df8bd-ad4f-46cb-be6d-bfa4e0e20e9c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:06:32.278736 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a9df8bd-ad4f-46cb-be6d-bfa4e0e20e9c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:06:32.280108 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a9df8bd-ad4f-46cb-be6d-bfa4e0e20e9c service neutron] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:06:32.298104 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 741/1467] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:06:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 304 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:06:34.211047 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3395d45d-607d-490c-9452-3a642410233f None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:06:34.211647 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3395d45d-607d-490c-9452-3a642410233f None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:06:34.211647 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3395d45d-607d-490c-9452-3a642410233f None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:06:34.212959 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 727/1468] 127.0.0.1 () {54 vars in 917 bytes} [Tue Aug 30 05:06:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:06:34.219159 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1805bf8-47dd-4a89-820c-18ee27ccf5cc None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:06:34.219605 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1805bf8-47dd-4a89-820c-18ee27ccf5cc None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:06:34.219718 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1805bf8-47dd-4a89-820c-18ee27ccf5cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:06:34.284233 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-f1805bf8-47dd-4a89-820c-18ee27ccf5cc None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:06:34.285422 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-f1805bf8-47dd-4a89-820c-18ee27ccf5cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:06:34.333675 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:06:34.498851 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 742/1469] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Aug 30 05:06:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 281 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:06:34.780878 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3395d45d-607d-490c-9452-3a642410233f None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:06:34.782416 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-3395d45d-607d-490c-9452-3a642410233f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:06:34.883761 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3395d45d-607d-490c-9452-3a642410233f None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:06:34.929590 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:06:34.956461 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc9a99eb-b503-4985-9723-e696aeadbfc0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:06:34.957057 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc9a99eb-b503-4985-9723-e696aeadbfc0 service neutron] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:06:34.957335 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc9a99eb-b503-4985-9723-e696aeadbfc0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:06:34.957602 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc9a99eb-b503-4985-9723-e696aeadbfc0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:06:34.959271 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-dc9a99eb-b503-4985-9723-e696aeadbfc0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:06:35.066462 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc9a99eb-b503-4985-9723-e696aeadbfc0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:06:35.067927 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc9a99eb-b503-4985-9723-e696aeadbfc0 service neutron] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:06:35.085899 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 728/1470] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:06:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 306 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:06:44.497028 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=22 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:06:45.083698 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=66 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:06:54.390933 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73b2619a-657a-43f5-a098-c6b00e4049be None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:06:54.391500 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73b2619a-657a-43f5-a098-c6b00e4049be None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:06:54.391500 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73b2619a-657a-43f5-a098-c6b00e4049be None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:06:54.392798 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 743/1471] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:06:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:06:54.398081 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d33a3f6-1663-4e40-b00f-cac95986f299 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:06:54.398511 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d33a3f6-1663-4e40-b00f-cac95986f299 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:06:54.398652 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d33a3f6-1663-4e40-b00f-cac95986f299 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:06:54.464564 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-3d33a3f6-1663-4e40-b00f-cac95986f299 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:06:54.466173 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-3d33a3f6-1663-4e40-b00f-cac95986f299 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:06:54.565978 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 729/1472] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:06:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:06:54.863555 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3035671-d550-42ff-b425-2850669675d7 None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:06:54.863834 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3035671-d550-42ff-b425-2850669675d7 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:06:54.864098 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3035671-d550-42ff-b425-2850669675d7 None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:06:54.865457 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 744/1473] 10.222.0.77 () {54 vars in 967 bytes} [Tue Aug 30 05:06:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:06:54.870169 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46ee51df-3a11-4dd6-8783-adfcbf2347ff None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:06:54.870461 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46ee51df-3a11-4dd6-8783-adfcbf2347ff None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:06:54.870737 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46ee51df-3a11-4dd6-8783-adfcbf2347ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:06:54.945602 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-46ee51df-3a11-4dd6-8783-adfcbf2347ff None None] MFA Rules not processed for user `e0a002ac0bf240a891ac7304178d3f0e`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:06:54.946829 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-46ee51df-3a11-4dd6-8783-adfcbf2347ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:06:54.967294 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:06:55.022057 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 730/1474] 10.222.0.77 () {58 vars in 1064 bytes} [Tue Aug 30 05:06:54 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:06:55.029435 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3035671-d550-42ff-b425-2850669675d7 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:06:55.031568 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-e3035671-d550-42ff-b425-2850669675d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:06:55.132282 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3035671-d550-42ff-b425-2850669675d7 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:06:55.207308 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36570f35-dc7e-4cf4-8d39-46def1d1f4b0 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a002ac0bf240a891ac7304178d3f0e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:06:55.207906 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36570f35-dc7e-4cf4-8d39-46def1d1f4b0 service glance] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:06:55.208181 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36570f35-dc7e-4cf4-8d39-46def1d1f4b0 service glance] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:06:55.208451 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36570f35-dc7e-4cf4-8d39-46def1d1f4b0 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:06:55.210027 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-36570f35-dc7e-4cf4-8d39-46def1d1f4b0 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:06:55.262959 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:06:55.317177 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36570f35-dc7e-4cf4-8d39-46def1d1f4b0 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:06:55.318625 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36570f35-dc7e-4cf4-8d39-46def1d1f4b0 service glance] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:06:55.336704 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 745/1475] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:06:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 308 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:06:57.325322 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9549d068-f2ff-491a-8448-49435c29ae44 None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:06:57.325933 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9549d068-f2ff-491a-8448-49435c29ae44 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:06:57.325933 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9549d068-f2ff-491a-8448-49435c29ae44 None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:06:57.327267 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 731/1476] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:06:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:06:57.332481 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cbefa6a-b07a-4e58-a826-7ca8d50b9a00 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:06:57.332910 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cbefa6a-b07a-4e58-a826-7ca8d50b9a00 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:06:57.333013 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cbefa6a-b07a-4e58-a826-7ca8d50b9a00 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:06:57.397390 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-1cbefa6a-b07a-4e58-a826-7ca8d50b9a00 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:06:57.398580 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-1cbefa6a-b07a-4e58-a826-7ca8d50b9a00 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:06:57.496934 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 746/1477] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:06:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:06:57.770962 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9549d068-f2ff-491a-8448-49435c29ae44 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:06:57.772505 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-9549d068-f2ff-491a-8448-49435c29ae44 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:06:57.897128 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9549d068-f2ff-491a-8448-49435c29ae44 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:06:57.945598 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:06:57.991130 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d53eaf88-3861-4dc4-bfcf-084f93331346 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:06:57.991730 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d53eaf88-3861-4dc4-bfcf-084f93331346 admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:06:57.992006 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d53eaf88-3861-4dc4-bfcf-084f93331346 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:06:57.992276 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d53eaf88-3861-4dc4-bfcf-084f93331346 admin admin] PATH_INFO: `/v3/projects` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:06:57.992906 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d53eaf88-3861-4dc4-bfcf-084f93331346 admin admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:06:57.994704 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: WARNING py.warnings [None req-d53eaf88-3861-4dc4-bfcf-084f93331346 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_projects": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 30 05:06:57.994704 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: warnings.warn(msg) Aug 30 05:06:57.994704 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: Aug 30 05:06:57.995880 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d53eaf88-3861-4dc4-bfcf-084f93331346 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:06:58.014061 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 732/1478] 10.222.0.77 () {56 vars in 1108 bytes} [Tue Aug 30 05:06:57 2022] GET /identity/v3/projects => generated 2553 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:06:59.915092 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-081ab7eb-331c-490b-885f-c105448084ed None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:06:59.915626 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-081ab7eb-331c-490b-885f-c105448084ed None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:06:59.915626 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-081ab7eb-331c-490b-885f-c105448084ed None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:06:59.916957 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 747/1479] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:06:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:06:59.922267 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96981735-bddc-4617-9262-e2fc38e91461 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:06:59.922683 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96981735-bddc-4617-9262-e2fc38e91461 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:06:59.922880 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96981735-bddc-4617-9262-e2fc38e91461 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:06:59.989063 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-96981735-bddc-4617-9262-e2fc38e91461 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:06:59.990261 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-96981735-bddc-4617-9262-e2fc38e91461 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:07:00.085688 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 733/1480] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:06:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:07:00.383452 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-081ab7eb-331c-490b-885f-c105448084ed None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:07:00.385102 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-081ab7eb-331c-490b-885f-c105448084ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:07:00.394912 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:07:00.529488 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-081ab7eb-331c-490b-885f-c105448084ed None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:07:00.610868 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:07:00.643295 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23fa8f25-dac0-4d14-bdbc-5418db8dc0bf service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '886eb5e00bbf4ec292eb962b665b37d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:07:00.643824 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23fa8f25-dac0-4d14-bdbc-5418db8dc0bf service nova] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:07:00.644098 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23fa8f25-dac0-4d14-bdbc-5418db8dc0bf service nova] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:07:00.644364 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23fa8f25-dac0-4d14-bdbc-5418db8dc0bf service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:07:00.645934 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-23fa8f25-dac0-4d14-bdbc-5418db8dc0bf service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:07:00.733053 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23fa8f25-dac0-4d14-bdbc-5418db8dc0bf service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:07:00.735315 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23fa8f25-dac0-4d14-bdbc-5418db8dc0bf service nova] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:07:00.752785 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 748/1481] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:07:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 370 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:07:02.814388 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af5281f6-1fd5-4071-af50-76fed5bab960 None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:07:02.814925 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af5281f6-1fd5-4071-af50-76fed5bab960 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:07:02.815034 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af5281f6-1fd5-4071-af50-76fed5bab960 None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:07:02.816315 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 734/1482] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:07:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:07:02.821485 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa8f642a-b8dd-4dfc-a18c-d063b8dc72ca None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:07:02.821956 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa8f642a-b8dd-4dfc-a18c-d063b8dc72ca None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:07:02.822053 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa8f642a-b8dd-4dfc-a18c-d063b8dc72ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:07:02.886728 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-fa8f642a-b8dd-4dfc-a18c-d063b8dc72ca None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:07:02.887951 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-fa8f642a-b8dd-4dfc-a18c-d063b8dc72ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:07:02.934110 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:07:02.985558 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 749/1483] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:07:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:07:03.268129 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af5281f6-1fd5-4071-af50-76fed5bab960 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:07:03.269672 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-af5281f6-1fd5-4071-af50-76fed5bab960 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:07:03.293733 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:07:03.375132 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af5281f6-1fd5-4071-af50-76fed5bab960 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:07:03.447524 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52497627-e8ba-4c3f-a947-c62f5c337390 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:07:03.448116 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52497627-e8ba-4c3f-a947-c62f5c337390 service neutron] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:07:03.448390 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52497627-e8ba-4c3f-a947-c62f5c337390 service neutron] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:07:03.448656 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52497627-e8ba-4c3f-a947-c62f5c337390 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:07:03.450219 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-52497627-e8ba-4c3f-a947-c62f5c337390 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:07:03.525198 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:07:03.560152 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52497627-e8ba-4c3f-a947-c62f5c337390 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:07:03.561587 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52497627-e8ba-4c3f-a947-c62f5c337390 service neutron] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:07:03.582322 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 735/1484] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:07:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 315 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:07:05.530166 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f638a6-0712-4e38-b886-79c5828410a5 None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:07:05.531406 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f638a6-0712-4e38-b886-79c5828410a5 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:07:05.531406 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f638a6-0712-4e38-b886-79c5828410a5 None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:07:05.532649 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 750/1485] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:07:05 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:07:05.537756 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21da0ef-cd4c-48ad-a6df-b89695886956 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:07:05.538174 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21da0ef-cd4c-48ad-a6df-b89695886956 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:07:05.538313 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21da0ef-cd4c-48ad-a6df-b89695886956 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:07:05.603804 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-d21da0ef-cd4c-48ad-a6df-b89695886956 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:07:05.605012 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-d21da0ef-cd4c-48ad-a6df-b89695886956 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:07:05.700774 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 736/1486] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:07:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:07:05.988885 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54f638a6-0712-4e38-b886-79c5828410a5 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:07:05.990421 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-54f638a6-0712-4e38-b886-79c5828410a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:07:06.093274 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54f638a6-0712-4e38-b886-79c5828410a5 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:07:06.163665 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:07:06.165572 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25510b23-2030-4828-b2ff-df5ab60bf067 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:07:06.166362 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25510b23-2030-4828-b2ff-df5ab60bf067 service neutron] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:07:06.166793 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25510b23-2030-4828-b2ff-df5ab60bf067 service neutron] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:07:06.167286 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25510b23-2030-4828-b2ff-df5ab60bf067 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:07:06.168912 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-25510b23-2030-4828-b2ff-df5ab60bf067 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:07:06.277668 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25510b23-2030-4828-b2ff-df5ab60bf067 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:07:06.279050 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25510b23-2030-4828-b2ff-df5ab60bf067 service neutron] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:07:06.297008 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 751/1487] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:07:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:07:08.283516 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27091a7e-36ed-4153-b214-96cb80cd200f None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:07:08.284059 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27091a7e-36ed-4153-b214-96cb80cd200f None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:07:08.284059 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27091a7e-36ed-4153-b214-96cb80cd200f None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:07:08.285433 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 737/1488] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:07:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:07:08.290761 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143ae7ae-3749-4c79-b63a-c9e59179eae3 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:07:08.291215 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143ae7ae-3749-4c79-b63a-c9e59179eae3 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:07:08.291343 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143ae7ae-3749-4c79-b63a-c9e59179eae3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:07:08.356587 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-143ae7ae-3749-4c79-b63a-c9e59179eae3 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:07:08.357799 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-143ae7ae-3749-4c79-b63a-c9e59179eae3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:07:08.412341 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:07:08.460083 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 752/1489] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:07:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:07:08.754969 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27091a7e-36ed-4153-b214-96cb80cd200f None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:07:08.756497 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-27091a7e-36ed-4153-b214-96cb80cd200f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:07:08.795958 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:07:08.863609 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27091a7e-36ed-4153-b214-96cb80cd200f None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:07:08.941092 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c1924ad-58d7-4a06-a764-afe2f0ea1747 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:07:08.941695 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c1924ad-58d7-4a06-a764-afe2f0ea1747 service neutron] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:07:08.941972 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c1924ad-58d7-4a06-a764-afe2f0ea1747 service neutron] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:07:08.942241 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c1924ad-58d7-4a06-a764-afe2f0ea1747 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:07:08.943920 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-7c1924ad-58d7-4a06-a764-afe2f0ea1747 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:07:09.024432 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:07:09.053351 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c1924ad-58d7-4a06-a764-afe2f0ea1747 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:07:09.054781 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c1924ad-58d7-4a06-a764-afe2f0ea1747 service neutron] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:07:09.190938 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 738/1490] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:07:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 437 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:07:11.260795 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c214759-a799-4875-ab10-7a8e113fb70d None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:07:11.261356 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c214759-a799-4875-ab10-7a8e113fb70d None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:07:11.261356 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c214759-a799-4875-ab10-7a8e113fb70d None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:07:11.262660 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 753/1491] 10.222.0.77 () {54 vars in 927 bytes} [Tue Aug 30 05:07:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:07:11.268358 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44f05e63-041f-48de-b938-4e9fe8b33733 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:07:11.268784 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44f05e63-041f-48de-b938-4e9fe8b33733 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:07:11.268901 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44f05e63-041f-48de-b938-4e9fe8b33733 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:07:11.333987 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-44f05e63-041f-48de-b938-4e9fe8b33733 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:07:11.335213 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-44f05e63-041f-48de-b938-4e9fe8b33733 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:07:11.429606 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 739/1492] 10.222.0.77 () {58 vars in 1024 bytes} [Tue Aug 30 05:07:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 05:07:11.718333 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c214759-a799-4875-ab10-7a8e113fb70d None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:07:11.719896 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-4c214759-a799-4875-ab10-7a8e113fb70d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:07:11.820985 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c214759-a799-4875-ab10-7a8e113fb70d None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:07:11.893381 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b178423a-d13b-4a45-8b42-e6195182336c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:07:11.893971 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b178423a-d13b-4a45-8b42-e6195182336c service neutron] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:07:11.894246 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b178423a-d13b-4a45-8b42-e6195182336c service neutron] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:07:11.894509 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b178423a-d13b-4a45-8b42-e6195182336c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:07:11.896093 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-b178423a-d13b-4a45-8b42-e6195182336c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:07:11.904934 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:07:12.004413 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b178423a-d13b-4a45-8b42-e6195182336c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:07:12.005864 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b178423a-d13b-4a45-8b42-e6195182336c service neutron] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:07:12.023440 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 754/1493] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:07:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 306 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:07:21.427855 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=82 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:07:22.021057 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=50 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:08:23.131755 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f8ddbff-46e9-4065-8ee2-7986beafc4f5 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:08:23.132345 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f8ddbff-46e9-4065-8ee2-7986beafc4f5 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:08:23.132345 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f8ddbff-46e9-4065-8ee2-7986beafc4f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:08:23.207237 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-9f8ddbff-46e9-4065-8ee2-7986beafc4f5 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:08:23.208424 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-9f8ddbff-46e9-4065-8ee2-7986beafc4f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:08:23.301902 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 740/1494] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:08:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:08:23.310114 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36e3fe57-a283-45a0-96bf-2acaedb4dc3b None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:08:23.310629 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36e3fe57-a283-45a0-96bf-2acaedb4dc3b None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:08:23.310629 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36e3fe57-a283-45a0-96bf-2acaedb4dc3b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:08:23.388148 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-36e3fe57-a283-45a0-96bf-2acaedb4dc3b None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:08:23.389290 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-36e3fe57-a283-45a0-96bf-2acaedb4dc3b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:08:23.485140 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 755/1495] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:08:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 05:08:23.489878 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f8ddbff-46e9-4065-8ee2-7986beafc4f5 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:08:23.491111 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-9f8ddbff-46e9-4065-8ee2-7986beafc4f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:08:23.503145 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f8ddbff-46e9-4065-8ee2-7986beafc4f5 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:08:23.564631 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:08:23.598712 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dce68d99-c238-45e8-8013-cae6b3997de6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:08:23.599338 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dce68d99-c238-45e8-8013-cae6b3997de6 admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:08:23.599614 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dce68d99-c238-45e8-8013-cae6b3997de6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:08:23.599882 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dce68d99-c238-45e8-8013-cae6b3997de6 admin admin] PATH_INFO: `/v3/domains` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:08:23.600513 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dce68d99-c238-45e8-8013-cae6b3997de6 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:08:23.600809 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dce68d99-c238-45e8-8013-cae6b3997de6 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:08:23.601814 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: WARNING py.warnings [None req-dce68d99-c238-45e8-8013-cae6b3997de6 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Aug 30 05:08:23.601814 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: warnings.warn(msg) Aug 30 05:08:23.601814 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: Aug 30 05:08:23.602690 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dce68d99-c238-45e8-8013-cae6b3997de6 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:08:23.614976 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 741/1496] 10.222.0.77 () {58 vars in 1153 bytes} [Tue Aug 30 05:08:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 126 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:08:23.623066 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36e3fe57-a283-45a0-96bf-2acaedb4dc3b None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:08:23.631915 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36e3fe57-a283-45a0-96bf-2acaedb4dc3b None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:08:23.694002 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:08:23.727993 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-716b754b-ef28-414d-ba6c-e495fc99f2cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:08:23.728581 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-716b754b-ef28-414d-ba6c-e495fc99f2cf admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:08:23.728851 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-716b754b-ef28-414d-ba6c-e495fc99f2cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:08:23.729109 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-716b754b-ef28-414d-ba6c-e495fc99f2cf admin admin] PATH_INFO: `/v3/projects` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:08:23.729730 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-716b754b-ef28-414d-ba6c-e495fc99f2cf admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:08:23.732053 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: WARNING py.warnings [None req-716b754b-ef28-414d-ba6c-e495fc99f2cf admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 30 05:08:23.732053 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: warnings.warn(msg) Aug 30 05:08:23.732053 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: Aug 30 05:08:23.733235 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-716b754b-ef28-414d-ba6c-e495fc99f2cf admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:08:23.788099 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 756/1497] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:08:23 2022] POST /identity/v3/projects => generated 377 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:08:23.792226 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dce68d99-c238-45e8-8013-cae6b3997de6 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:08:23.794322 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-dce68d99-c238-45e8-8013-cae6b3997de6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:08:23.826550 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dce68d99-c238-45e8-8013-cae6b3997de6 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:08:23.918126 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a09192e-3120-4112-b6b7-3d975a35fa81 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:08:23.919392 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a09192e-3120-4112-b6b7-3d975a35fa81 admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:08:23.919669 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a09192e-3120-4112-b6b7-3d975a35fa81 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:08:23.919936 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a09192e-3120-4112-b6b7-3d975a35fa81 admin admin] PATH_INFO: `/v3/users` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:08:23.920567 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a09192e-3120-4112-b6b7-3d975a35fa81 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:08:23.921592 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: WARNING py.warnings [None req-7a09192e-3120-4112-b6b7-3d975a35fa81 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 30 05:08:23.921592 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: warnings.warn(msg) Aug 30 05:08:23.921592 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: Aug 30 05:08:23.922796 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a09192e-3120-4112-b6b7-3d975a35fa81 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:08:23.963992 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 742/1498] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:08:23 2022] POST /identity/v3/users => generated 289 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:08:23.968464 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-716b754b-ef28-414d-ba6c-e495fc99f2cf admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:08:23.969993 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-716b754b-ef28-414d-ba6c-e495fc99f2cf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:08:24.032410 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-716b754b-ef28-414d-ba6c-e495fc99f2cf admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:08:24.114831 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:08:24.126698 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be1d7bad-a468-488d-81b5-5e18d72da1a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:08:24.127321 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be1d7bad-a468-488d-81b5-5e18d72da1a0 admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:08:24.127605 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be1d7bad-a468-488d-81b5-5e18d72da1a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:08:24.127870 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be1d7bad-a468-488d-81b5-5e18d72da1a0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:08:24.128497 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be1d7bad-a468-488d-81b5-5e18d72da1a0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:08:24.129498 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: WARNING py.warnings [None req-be1d7bad-a468-488d-81b5-5e18d72da1a0 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Aug 30 05:08:24.129498 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: warnings.warn(msg) Aug 30 05:08:24.129498 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: Aug 30 05:08:24.130376 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be1d7bad-a468-488d-81b5-5e18d72da1a0 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:08:24.142504 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 757/1499] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:08:23 2022] GET /identity/v3/roles => generated 1420 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:08:24.146499 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a09192e-3120-4112-b6b7-3d975a35fa81 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:08:24.148047 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-7a09192e-3120-4112-b6b7-3d975a35fa81 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:08:24.159369 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a09192e-3120-4112-b6b7-3d975a35fa81 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:08:24.185312 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:08:24.255379 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55d0b84f-dabf-4d38-bb2e-92bb4ef498fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:08:24.255909 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55d0b84f-dabf-4d38-bb2e-92bb4ef498fc admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:08:24.256188 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55d0b84f-dabf-4d38-bb2e-92bb4ef498fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:08:24.256453 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55d0b84f-dabf-4d38-bb2e-92bb4ef498fc admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:08:24.257077 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55d0b84f-dabf-4d38-bb2e-92bb4ef498fc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:08:24.258077 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: WARNING py.warnings [None req-55d0b84f-dabf-4d38-bb2e-92bb4ef498fc admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Aug 30 05:08:24.258077 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: warnings.warn(msg) Aug 30 05:08:24.258077 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: Aug 30 05:08:24.258977 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55d0b84f-dabf-4d38-bb2e-92bb4ef498fc admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:08:24.271709 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 743/1500] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:08:24 2022] GET /identity/v3/roles => generated 1420 bytes in 126 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:08:24.276235 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be1d7bad-a468-488d-81b5-5e18d72da1a0 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:08:24.277745 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-be1d7bad-a468-488d-81b5-5e18d72da1a0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:08:24.311093 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be1d7bad-a468-488d-81b5-5e18d72da1a0 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:08:24.404441 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b5265d8-7596-472e-a32d-afc52ef67a71 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:08:24.405083 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b5265d8-7596-472e-a32d-afc52ef67a71 admin admin] REQUEST_METHOD: `PUT` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:08:24.405358 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b5265d8-7596-472e-a32d-afc52ef67a71 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:08:24.405625 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b5265d8-7596-472e-a32d-afc52ef67a71 admin admin] PATH_INFO: `/v3/projects/7127c7ed6f114ca98880a7041bc0ae49/users/bf52f089a8c84330ae8f067acb6c6b8d/roles/6f03d86980c34c36b44a8d6764ff4fda` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:08:24.427707 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b5265d8-7596-472e-a32d-afc52ef67a71 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7127c7ed6f114ca98880a7041bc0ae49, user_id=bf52f089a8c84330ae8f067acb6c6b8d, role_id=6f03d86980c34c36b44a8d6764ff4fda)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:08:24.428769 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: WARNING py.warnings [None req-9b5265d8-7596-472e-a32d-afc52ef67a71 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 30 05:08:24.428769 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: warnings.warn(msg) Aug 30 05:08:24.428769 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: Aug 30 05:08:24.430713 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b5265d8-7596-472e-a32d-afc52ef67a71 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:08:24.454768 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 758/1501] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:08:24 2022] PUT /identity/v3/projects/7127c7ed6f114ca98880a7041bc0ae49/users/bf52f089a8c84330ae8f067acb6c6b8d/roles/6f03d86980c34c36b44a8d6764ff4fda => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:08:24.467293 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff1759b6-8240-476c-8404-7f9189391d07 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:08:24.467585 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff1759b6-8240-476c-8404-7f9189391d07 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:08:24.467858 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff1759b6-8240-476c-8404-7f9189391d07 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:08:24.523279 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-ff1759b6-8240-476c-8404-7f9189391d07 None None] MFA Rules not processed for user `bf52f089a8c84330ae8f067acb6c6b8d`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:08:24.524411 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-ff1759b6-8240-476c-8404-7f9189391d07 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:08:24.601078 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 744/1502] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:08:24 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 05:08:24.614499 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fcfb6bd-aafd-4b72-af04-b9635be7754f None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:08:24.614785 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fcfb6bd-aafd-4b72-af04-b9635be7754f None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:08:24.615080 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fcfb6bd-aafd-4b72-af04-b9635be7754f None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:08:24.616384 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 759/1503] 10.222.0.77 () {54 vars in 964 bytes} [Tue Aug 30 05:08:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:08:24.621308 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3769401e-810e-4be7-a00b-1032e0781874 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:08:24.621826 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3769401e-810e-4be7-a00b-1032e0781874 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:08:24.622101 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3769401e-810e-4be7-a00b-1032e0781874 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:08:24.654510 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:08:24.698544 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-3769401e-810e-4be7-a00b-1032e0781874 None None] MFA Rules not processed for user `019a79b2affe478b8a30d1c538d0415c`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:08:24.699759 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-3769401e-810e-4be7-a00b-1032e0781874 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:08:24.773527 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 745/1504] 10.222.0.77 () {58 vars in 1061 bytes} [Tue Aug 30 05:08:24 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:08:24.781821 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fcfb6bd-aafd-4b72-af04-b9635be7754f None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:08:24.783327 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-3fcfb6bd-aafd-4b72-af04-b9635be7754f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:08:24.832443 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:08:24.886691 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fcfb6bd-aafd-4b72-af04-b9635be7754f None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:08:24.964567 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-194d5143-fbd3-4e82-8450-3a733f3ea776 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '019a79b2affe478b8a30d1c538d0415c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:08:24.965384 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194d5143-fbd3-4e82-8450-3a733f3ea776 service cinder] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:08:24.966259 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194d5143-fbd3-4e82-8450-3a733f3ea776 service cinder] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:08:24.966546 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194d5143-fbd3-4e82-8450-3a733f3ea776 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:08:24.968320 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-194d5143-fbd3-4e82-8450-3a733f3ea776 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:08:25.065217 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-194d5143-fbd3-4e82-8450-3a733f3ea776 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:08:25.066637 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-194d5143-fbd3-4e82-8450-3a733f3ea776 service cinder] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:08:25.070038 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:08:25.085128 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 760/1505] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:08:24 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 304 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:08:25.129779 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5de550a-530b-483c-a819-8ed868bdd037 None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:08:25.130066 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5de550a-530b-483c-a819-8ed868bdd037 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:08:25.130339 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5de550a-530b-483c-a819-8ed868bdd037 None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:08:25.131912 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 746/1506] 10.222.0.77 () {52 vars in 830 bytes} [Tue Aug 30 05:08:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:08:25.135462 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-194d5143-fbd3-4e82-8450-3a733f3ea776 service cinder] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:08:25.136659 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-194d5143-fbd3-4e82-8450-3a733f3ea776 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:08:25.235726 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-194d5143-fbd3-4e82-8450-3a733f3ea776 service cinder] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:08:25.327643 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-312d0f2f-0d38-495e-ad83-33ef45226a65 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:08:25.328249 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-312d0f2f-0d38-495e-ad83-33ef45226a65 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:08:25.328440 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-312d0f2f-0d38-495e-ad83-33ef45226a65 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:08:25.328704 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-312d0f2f-0d38-495e-ad83-33ef45226a65 admin admin] PATH_INFO: `/v3/users/bf52f089a8c84330ae8f067acb6c6b8d` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:08:25.340414 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-312d0f2f-0d38-495e-ad83-33ef45226a65 admin admin] RBAC: Authorizing `identity:delete_user(user_id=bf52f089a8c84330ae8f067acb6c6b8d)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:08:25.341420 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: WARNING py.warnings [None req-312d0f2f-0d38-495e-ad83-33ef45226a65 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 30 05:08:25.341420 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: warnings.warn(msg) Aug 30 05:08:25.341420 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: Aug 30 05:08:25.342618 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-312d0f2f-0d38-495e-ad83-33ef45226a65 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:08:25.352183 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:08:25.407906 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-312d0f2f-0d38-495e-ad83-33ef45226a65 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bf52f089a8c84330ae8f067acb6c6b8d'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:08:25.417243 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-312d0f2f-0d38-495e-ad83-33ef45226a65 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bf52f089a8c84330ae8f067acb6c6b8d'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:08:25.439785 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-312d0f2f-0d38-495e-ad83-33ef45226a65 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bf52f089a8c84330ae8f067acb6c6b8d'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:08:25.454192 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-312d0f2f-0d38-495e-ad83-33ef45226a65 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bf52f089a8c84330ae8f067acb6c6b8d'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:08:25.454650 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-312d0f2f-0d38-495e-ad83-33ef45226a65 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bf52f089a8c84330ae8f067acb6c6b8d'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:08:25.460403 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 761/1507] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:08:25 2022] DELETE /identity/v3/users/bf52f089a8c84330ae8f067acb6c6b8d => generated 0 bytes in 325 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:08:25.464737 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2db0abc-b3db-4984-9dd6-83c9ee420400 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:08:25.465234 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2db0abc-b3db-4984-9dd6-83c9ee420400 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:08:25.465331 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2db0abc-b3db-4984-9dd6-83c9ee420400 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:08:25.521160 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-e2db0abc-b3db-4984-9dd6-83c9ee420400 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:08:25.522360 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-e2db0abc-b3db-4984-9dd6-83c9ee420400 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:08:25.596814 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:08:25.610360 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 747/1508] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:08:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:08:25.619948 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-312d0f2f-0d38-495e-ad83-33ef45226a65 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:08:25.621159 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-312d0f2f-0d38-495e-ad83-33ef45226a65 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:08:25.720835 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-312d0f2f-0d38-495e-ad83-33ef45226a65 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:08:25.774806 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:08:25.795309 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7c986a0-7d25-41f2-9278-0dfadea03a04 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:08:25.795905 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7c986a0-7d25-41f2-9278-0dfadea03a04 service neutron] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:08:25.796180 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7c986a0-7d25-41f2-9278-0dfadea03a04 service neutron] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:08:25.796444 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7c986a0-7d25-41f2-9278-0dfadea03a04 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:08:25.798001 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-d7c986a0-7d25-41f2-9278-0dfadea03a04 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:08:25.902664 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7c986a0-7d25-41f2-9278-0dfadea03a04 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:08:25.904529 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7c986a0-7d25-41f2-9278-0dfadea03a04 service neutron] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:08:25.922870 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 762/1509] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:08:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 303 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:08:26.251738 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2db0abc-b3db-4984-9dd6-83c9ee420400 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:08:26.252969 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-e2db0abc-b3db-4984-9dd6-83c9ee420400 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:08:26.264909 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2db0abc-b3db-4984-9dd6-83c9ee420400 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:08:26.360776 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22780ce1-da5b-4746-8339-e036fd60ba6f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:08:26.361386 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22780ce1-da5b-4746-8339-e036fd60ba6f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:08:26.361658 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22780ce1-da5b-4746-8339-e036fd60ba6f admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:08:26.361921 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22780ce1-da5b-4746-8339-e036fd60ba6f admin admin] PATH_INFO: `/v3/projects/7127c7ed6f114ca98880a7041bc0ae49` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:08:26.371573 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22780ce1-da5b-4746-8339-e036fd60ba6f admin admin] RBAC: Authorizing `identity:delete_project(project_id=7127c7ed6f114ca98880a7041bc0ae49)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:08:26.372587 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: WARNING py.warnings [None req-22780ce1-da5b-4746-8339-e036fd60ba6f admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 30 05:08:26.372587 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: warnings.warn(msg) Aug 30 05:08:26.372587 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: Aug 30 05:08:26.373775 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22780ce1-da5b-4746-8339-e036fd60ba6f admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:08:26.431412 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-22780ce1-da5b-4746-8339-e036fd60ba6f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7127c7ed6f114ca98880a7041bc0ae49', 'request_id': 'req-22780ce1-da5b-4746-8339-e036fd60ba6f'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:08:26.441817 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-22780ce1-da5b-4746-8339-e036fd60ba6f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7127c7ed6f114ca98880a7041bc0ae49', 'request_id': 'req-22780ce1-da5b-4746-8339-e036fd60ba6f'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:08:26.455855 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-22780ce1-da5b-4746-8339-e036fd60ba6f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7127c7ed6f114ca98880a7041bc0ae49', 'request_id': 'req-22780ce1-da5b-4746-8339-e036fd60ba6f'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:08:26.479117 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-22780ce1-da5b-4746-8339-e036fd60ba6f admin admin] The token cache is being invalidate because project 7127c7ed6f114ca98880a7041bc0ae49 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81624) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:08:26.479712 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-22780ce1-da5b-4746-8339-e036fd60ba6f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:08:26.479909 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-22780ce1-da5b-4746-8339-e036fd60ba6f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:08:26.481179 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 748/1510] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:08:26 2022] DELETE /identity/v3/projects/7127c7ed6f114ca98880a7041bc0ae49 => generated 0 bytes in 230 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:08:35.920618 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=62 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:08:36.477091 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=84,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:09:52.786311 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b3546b7-3102-45ca-a1ce-da2978f667ea None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:09:52.786903 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b3546b7-3102-45ca-a1ce-da2978f667ea None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:09:52.786903 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b3546b7-3102-45ca-a1ce-da2978f667ea None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:09:52.788145 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 763/1511] 127.0.0.1 () {54 vars in 901 bytes} [Tue Aug 30 05:09:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:09:52.792878 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb0f3014-1bf7-4f35-8014-5ae7d0912686 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:09:52.793317 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb0f3014-1bf7-4f35-8014-5ae7d0912686 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:09:52.793474 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb0f3014-1bf7-4f35-8014-5ae7d0912686 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:09:52.875884 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-fb0f3014-1bf7-4f35-8014-5ae7d0912686 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:09:52.877090 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-fb0f3014-1bf7-4f35-8014-5ae7d0912686 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:09:52.982703 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 749/1512] 127.0.0.1 () {58 vars in 998 bytes} [Tue Aug 30 05:09:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:09:53.136432 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b3546b7-3102-45ca-a1ce-da2978f667ea None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:09:53.138607 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-8b3546b7-3102-45ca-a1ce-da2978f667ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:09:53.265216 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b3546b7-3102-45ca-a1ce-da2978f667ea None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:09:53.271238 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba67d753-2ab6-464f-becb-9d40c58bff51 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '886eb5e00bbf4ec292eb962b665b37d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:09:53.271835 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba67d753-2ab6-464f-becb-9d40c58bff51 service nova] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:09:53.272122 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba67d753-2ab6-464f-becb-9d40c58bff51 service nova] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:09:53.272395 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba67d753-2ab6-464f-becb-9d40c58bff51 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:09:53.280350 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba67d753-2ab6-464f-becb-9d40c58bff51 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:09:53.282593 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba67d753-2ab6-464f-becb-9d40c58bff51 service nova] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:09:53.291863 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 764/1513] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:09:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 156 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:09:54.124301 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c36c560-7dae-4ea7-ad9e-c24f39c7d860 None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:09:54.124868 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c36c560-7dae-4ea7-ad9e-c24f39c7d860 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:09:54.124868 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c36c560-7dae-4ea7-ad9e-c24f39c7d860 None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:09:54.126783 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 750/1514] 127.0.0.1 () {54 vars in 901 bytes} [Tue Aug 30 05:09:54 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:09:54.131503 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-682d5950-7466-4227-be30-9915da8bc509 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:09:54.131947 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-682d5950-7466-4227-be30-9915da8bc509 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:09:54.132060 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-682d5950-7466-4227-be30-9915da8bc509 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:09:54.151163 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-682d5950-7466-4227-be30-9915da8bc509 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:09:54.152351 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-682d5950-7466-4227-be30-9915da8bc509 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:09:54.160017 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 765/1515] 127.0.0.1 () {58 vars in 998 bytes} [Tue Aug 30 05:09:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 05:09:54.313249 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c36c560-7dae-4ea7-ad9e-c24f39c7d860 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:09:54.328258 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c36c560-7dae-4ea7-ad9e-c24f39c7d860 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:09:54.333617 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51b17ed6-e8f2-4c9c-9481-db95f01dbdc3 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '886eb5e00bbf4ec292eb962b665b37d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:09:54.334204 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51b17ed6-e8f2-4c9c-9481-db95f01dbdc3 service nova] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:09:54.334478 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51b17ed6-e8f2-4c9c-9481-db95f01dbdc3 service nova] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:09:54.334747 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51b17ed6-e8f2-4c9c-9481-db95f01dbdc3 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:09:54.342706 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51b17ed6-e8f2-4c9c-9481-db95f01dbdc3 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:09:54.345075 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51b17ed6-e8f2-4c9c-9481-db95f01dbdc3 service nova] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:09:54.354844 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 751/1516] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:09:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:09:55.135046 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3731a976-998d-4ab2-abd4-c905d8378ba3 None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:09:55.135629 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3731a976-998d-4ab2-abd4-c905d8378ba3 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:09:55.135629 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3731a976-998d-4ab2-abd4-c905d8378ba3 None None] PATH_INFO: `/v3` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:09:55.136834 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 766/1517] 10.222.0.82 () {54 vars in 925 bytes} [Tue Aug 30 05:09:55 2022] GET /identity/v3 => generated 255 bytes in 3 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:09:55.143386 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e56c1280-6fd0-475c-871e-dcbc13c062c3 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:09:55.143838 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e56c1280-6fd0-475c-871e-dcbc13c062c3 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:09:55.143944 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e56c1280-6fd0-475c-871e-dcbc13c062c3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:09:55.204667 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-e56c1280-6fd0-475c-871e-dcbc13c062c3 None None] MFA Rules not processed for user `1972959402704b89925001cd8d90cb0e`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:09:55.205893 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-e56c1280-6fd0-475c-871e-dcbc13c062c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:09:55.224458 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:09:55.278662 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 752/1518] 10.222.0.82 () {58 vars in 1015 bytes} [Tue Aug 30 05:09:55 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:09:59.079770 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69b202a5-92e7-4d8d-b305-7926ca9d8321 None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:09:59.080278 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69b202a5-92e7-4d8d-b305-7926ca9d8321 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:09:59.080278 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69b202a5-92e7-4d8d-b305-7926ca9d8321 None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:09:59.081668 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 767/1519] 10.222.0.77 () {54 vars in 961 bytes} [Tue Aug 30 05:09:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:09:59.086463 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf2234a-3e03-4cce-9f2b-f0d064981fe4 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:09:59.086907 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf2234a-3e03-4cce-9f2b-f0d064981fe4 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:09:59.087054 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf2234a-3e03-4cce-9f2b-f0d064981fe4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:09:59.107014 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-ecf2234a-3e03-4cce-9f2b-f0d064981fe4 None None] MFA Rules not processed for user `1972959402704b89925001cd8d90cb0e`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:09:59.108205 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-ecf2234a-3e03-4cce-9f2b-f0d064981fe4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:09:59.116597 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 753/1520] 10.222.0.77 () {58 vars in 1058 bytes} [Tue Aug 30 05:09:59 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:09:59.123760 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69b202a5-92e7-4d8d-b305-7926ca9d8321 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:09:59.136778 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69b202a5-92e7-4d8d-b305-7926ca9d8321 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:09:59.142224 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8101575c-74af-4b1c-ab52-38478d98579b service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1972959402704b89925001cd8d90cb0e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:09:59.142796 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8101575c-74af-4b1c-ab52-38478d98579b service placement] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:09:59.143099 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8101575c-74af-4b1c-ab52-38478d98579b service placement] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:09:59.143363 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8101575c-74af-4b1c-ab52-38478d98579b service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:09:59.151315 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8101575c-74af-4b1c-ab52-38478d98579b service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:09:59.152874 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8101575c-74af-4b1c-ab52-38478d98579b service placement] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:09:59.162794 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 768/1521] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:09:59 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 05:10:05.564837 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9710534-dc3e-4d99-bbba-90a0acb2946e None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:10:05.565391 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9710534-dc3e-4d99-bbba-90a0acb2946e None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:10:05.565391 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9710534-dc3e-4d99-bbba-90a0acb2946e None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:10:05.566762 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 754/1522] 127.0.0.1 () {54 vars in 901 bytes} [Tue Aug 30 05:10:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:10:05.571410 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e14115f-91bc-441f-9b2a-d8ef666eff48 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:10:05.571818 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e14115f-91bc-441f-9b2a-d8ef666eff48 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:10:05.571955 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e14115f-91bc-441f-9b2a-d8ef666eff48 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:10:05.590859 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-2e14115f-91bc-441f-9b2a-d8ef666eff48 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:10:05.592032 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-2e14115f-91bc-441f-9b2a-d8ef666eff48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:10:05.600513 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 769/1523] 127.0.0.1 () {58 vars in 998 bytes} [Tue Aug 30 05:10:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:10:05.753617 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9710534-dc3e-4d99-bbba-90a0acb2946e None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:10:05.767659 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9710534-dc3e-4d99-bbba-90a0acb2946e None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:10:05.772981 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18b17e30-cfcc-4721-b5dd-942be74b9545 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '886eb5e00bbf4ec292eb962b665b37d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:10:05.773557 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18b17e30-cfcc-4721-b5dd-942be74b9545 service nova] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:10:05.773833 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18b17e30-cfcc-4721-b5dd-942be74b9545 service nova] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:10:05.774098 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18b17e30-cfcc-4721-b5dd-942be74b9545 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:10:05.781878 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18b17e30-cfcc-4721-b5dd-942be74b9545 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:10:05.783951 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18b17e30-cfcc-4721-b5dd-942be74b9545 service nova] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:10:05.793113 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 755/1524] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:10:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:10:06.604159 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf21379b-7518-4903-94bd-3c8231ed46d2 None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:10:06.604608 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf21379b-7518-4903-94bd-3c8231ed46d2 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:10:06.604723 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf21379b-7518-4903-94bd-3c8231ed46d2 None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:10:06.606049 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 770/1525] 127.0.0.1 () {54 vars in 901 bytes} [Tue Aug 30 05:10:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:10:06.611191 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f56a0cd-2ade-4a96-90a8-e302d37d260f None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:10:06.611601 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f56a0cd-2ade-4a96-90a8-e302d37d260f None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:10:06.611745 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f56a0cd-2ade-4a96-90a8-e302d37d260f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:10:06.632753 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-5f56a0cd-2ade-4a96-90a8-e302d37d260f None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:10:06.633953 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-5f56a0cd-2ade-4a96-90a8-e302d37d260f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:10:06.641797 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 756/1526] 127.0.0.1 () {58 vars in 998 bytes} [Tue Aug 30 05:10:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:10:06.793017 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf21379b-7518-4903-94bd-3c8231ed46d2 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:10:06.807632 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf21379b-7518-4903-94bd-3c8231ed46d2 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:10:06.813034 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83c1f301-afdc-4177-beb1-a8bcafcbeb34 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '886eb5e00bbf4ec292eb962b665b37d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:10:06.814168 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83c1f301-afdc-4177-beb1-a8bcafcbeb34 service nova] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:10:06.814447 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83c1f301-afdc-4177-beb1-a8bcafcbeb34 service nova] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:10:06.814771 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83c1f301-afdc-4177-beb1-a8bcafcbeb34 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:10:06.822683 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83c1f301-afdc-4177-beb1-a8bcafcbeb34 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:10:06.824284 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83c1f301-afdc-4177-beb1-a8bcafcbeb34 service nova] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:10:06.833522 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 771/1527] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:10:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:10:10.607328 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae79cd3b-2609-45a2-9e29-46967208bb7b None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:10:10.607908 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae79cd3b-2609-45a2-9e29-46967208bb7b None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:10:10.607908 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae79cd3b-2609-45a2-9e29-46967208bb7b None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:10:10.609223 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 757/1528] 127.0.0.1 () {54 vars in 917 bytes} [Tue Aug 30 05:10:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:10:10.614742 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13acf8f7-2827-47b1-bd92-cf43261e28eb None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:10:10.615184 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13acf8f7-2827-47b1-bd92-cf43261e28eb None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:10:10.615321 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13acf8f7-2827-47b1-bd92-cf43261e28eb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:10:10.635208 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-13acf8f7-2827-47b1-bd92-cf43261e28eb None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:10:10.636398 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-13acf8f7-2827-47b1-bd92-cf43261e28eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:10:10.644103 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 772/1529] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Aug 30 05:10:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:10:10.928544 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae79cd3b-2609-45a2-9e29-46967208bb7b None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:10:10.930202 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-ae79cd3b-2609-45a2-9e29-46967208bb7b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:10:11.006916 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae79cd3b-2609-45a2-9e29-46967208bb7b None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:10:11.010421 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee32209f-7f52-443b-9b68-2aa46491a831 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:10:11.011038 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee32209f-7f52-443b-9b68-2aa46491a831 service neutron] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:10:11.011314 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee32209f-7f52-443b-9b68-2aa46491a831 service neutron] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:10:11.011578 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee32209f-7f52-443b-9b68-2aa46491a831 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:10:11.020519 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee32209f-7f52-443b-9b68-2aa46491a831 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:10:11.021982 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee32209f-7f52-443b-9b68-2aa46491a831 service neutron] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:10:11.031150 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 758/1530] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:10:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:10:12.897587 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d9050ac-8cd3-4187-bfd2-5668eb9b5689 None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:10:12.898147 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d9050ac-8cd3-4187-bfd2-5668eb9b5689 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:10:12.898147 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d9050ac-8cd3-4187-bfd2-5668eb9b5689 None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:10:12.899485 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 773/1531] 127.0.0.1 () {54 vars in 917 bytes} [Tue Aug 30 05:10:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:10:12.904886 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd5eec7-63c6-429e-8e48-f4005f013d29 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:10:12.905282 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd5eec7-63c6-429e-8e48-f4005f013d29 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:10:12.905446 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd5eec7-63c6-429e-8e48-f4005f013d29 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:10:12.926067 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-cfd5eec7-63c6-429e-8e48-f4005f013d29 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:10:12.927404 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-cfd5eec7-63c6-429e-8e48-f4005f013d29 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:10:12.935342 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 759/1532] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Aug 30 05:10:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:10:13.222771 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d9050ac-8cd3-4187-bfd2-5668eb9b5689 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:10:13.233663 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d9050ac-8cd3-4187-bfd2-5668eb9b5689 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:10:13.237176 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6773231b-da9b-4a22-95d3-07b16b646c36 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:10:13.237762 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6773231b-da9b-4a22-95d3-07b16b646c36 service neutron] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:10:13.238046 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6773231b-da9b-4a22-95d3-07b16b646c36 service neutron] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:10:13.238313 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6773231b-da9b-4a22-95d3-07b16b646c36 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:10:13.251140 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6773231b-da9b-4a22-95d3-07b16b646c36 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:10:13.252565 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6773231b-da9b-4a22-95d3-07b16b646c36 service neutron] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:10:13.261841 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 774/1533] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:10:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:10:16.556765 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8307874d-2450-41d3-a05f-6890b6719730 None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:10:16.557219 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8307874d-2450-41d3-a05f-6890b6719730 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:10:16.557315 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8307874d-2450-41d3-a05f-6890b6719730 None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:10:16.558662 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 760/1534] 127.0.0.1 () {54 vars in 917 bytes} [Tue Aug 30 05:10:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:10:16.563855 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f11899-474a-4334-810b-d0ad5d4cd7f4 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:10:16.564252 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f11899-474a-4334-810b-d0ad5d4cd7f4 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:10:16.564407 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f11899-474a-4334-810b-d0ad5d4cd7f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:10:16.572133 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=81 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:10:16.585106 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-49f11899-474a-4334-810b-d0ad5d4cd7f4 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:10:16.586303 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-49f11899-474a-4334-810b-d0ad5d4cd7f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:10:16.594041 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 775/1535] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Aug 30 05:10:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:10:17.142424 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8307874d-2450-41d3-a05f-6890b6719730 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:10:17.144081 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-8307874d-2450-41d3-a05f-6890b6719730 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:10:17.207840 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:10:17.221716 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8307874d-2450-41d3-a05f-6890b6719730 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:10:17.225416 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09ea2f24-3df5-495d-a6cb-adc2fa49fcea service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '019a79b2affe478b8a30d1c538d0415c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:10:17.226016 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ea2f24-3df5-495d-a6cb-adc2fa49fcea service cinder] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:10:17.226294 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ea2f24-3df5-495d-a6cb-adc2fa49fcea service cinder] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:10:17.226565 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ea2f24-3df5-495d-a6cb-adc2fa49fcea service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:10:17.235436 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09ea2f24-3df5-495d-a6cb-adc2fa49fcea service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:10:17.237511 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09ea2f24-3df5-495d-a6cb-adc2fa49fcea service cinder] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:10:17.246694 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 761/1536] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:10:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:10:19.154773 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccbc4a0d-f6ea-4afa-8396-13d58b0b75c4 None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:10:19.155379 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccbc4a0d-f6ea-4afa-8396-13d58b0b75c4 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:10:19.155379 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccbc4a0d-f6ea-4afa-8396-13d58b0b75c4 None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:10:19.156676 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 776/1537] 127.0.0.1 () {54 vars in 917 bytes} [Tue Aug 30 05:10:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:10:19.162564 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a044501b-f7a0-4f93-b3f3-52912b59502e None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:10:19.163026 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a044501b-f7a0-4f93-b3f3-52912b59502e None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:10:19.163150 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a044501b-f7a0-4f93-b3f3-52912b59502e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:10:19.182463 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-a044501b-f7a0-4f93-b3f3-52912b59502e None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:10:19.183711 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-a044501b-f7a0-4f93-b3f3-52912b59502e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:10:19.191646 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 762/1538] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Aug 30 05:10:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:10:19.757828 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccbc4a0d-f6ea-4afa-8396-13d58b0b75c4 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:10:19.759527 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-ccbc4a0d-f6ea-4afa-8396-13d58b0b75c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:10:19.772920 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccbc4a0d-f6ea-4afa-8396-13d58b0b75c4 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:10:19.776444 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d3eb9e8-5cdd-4ae7-bfde-a5cf5d0b550a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '019a79b2affe478b8a30d1c538d0415c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:10:19.777028 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d3eb9e8-5cdd-4ae7-bfde-a5cf5d0b550a service cinder] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:10:19.777303 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d3eb9e8-5cdd-4ae7-bfde-a5cf5d0b550a service cinder] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:10:19.777571 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d3eb9e8-5cdd-4ae7-bfde-a5cf5d0b550a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:10:19.786901 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d3eb9e8-5cdd-4ae7-bfde-a5cf5d0b550a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:10:19.788308 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d3eb9e8-5cdd-4ae7-bfde-a5cf5d0b550a service cinder] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:10:19.797705 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 777/1539] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:10:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:10:29.177372 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=11 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:10:29.793436 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=8 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:11:44.231671 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b341530c-4161-4969-9f44-a7b24aed3758 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:44.232808 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b341530c-4161-4969-9f44-a7b24aed3758 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:44.233318 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b341530c-4161-4969-9f44-a7b24aed3758 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:44.261353 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-b341530c-4161-4969-9f44-a7b24aed3758 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:11:44.263065 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-b341530c-4161-4969-9f44-a7b24aed3758 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:11:44.274102 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 763/1540] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:11:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 43 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 05:11:44.283276 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55495d72-d934-4059-be58-2c8337125688 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:44.285980 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55495d72-d934-4059-be58-2c8337125688 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:44.285980 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55495d72-d934-4059-be58-2c8337125688 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:44.312247 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-55495d72-d934-4059-be58-2c8337125688 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:11:44.313865 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-55495d72-d934-4059-be58-2c8337125688 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:11:44.326917 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 778/1541] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:11:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 45 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 05:11:44.332042 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b341530c-4161-4969-9f44-a7b24aed3758 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:44.344235 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b341530c-4161-4969-9f44-a7b24aed3758 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:44.351685 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d42e52d4-7023-4f87-bf27-b56ef72710e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:44.352641 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d42e52d4-7023-4f87-bf27-b56ef72710e0 admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:44.353370 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d42e52d4-7023-4f87-bf27-b56ef72710e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:44.353859 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d42e52d4-7023-4f87-bf27-b56ef72710e0 admin admin] PATH_INFO: `/v3/domains` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:44.354716 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d42e52d4-7023-4f87-bf27-b56ef72710e0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:11:44.355267 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d42e52d4-7023-4f87-bf27-b56ef72710e0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:44.357090 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d42e52d4-7023-4f87-bf27-b56ef72710e0 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:44.375104 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 764/1542] 10.222.0.77 () {58 vars in 1153 bytes} [Tue Aug 30 05:11:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:11:44.380553 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4905d2e9-a30f-444c-8fc6-a41f68cfdce7 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:44.381107 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4905d2e9-a30f-444c-8fc6-a41f68cfdce7 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:44.381598 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4905d2e9-a30f-444c-8fc6-a41f68cfdce7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:44.394591 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf470719-9a0b-4c93-afad-664c1129d5df None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:44.396478 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf470719-9a0b-4c93-afad-664c1129d5df None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:44.407122 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf470719-9a0b-4c93-afad-664c1129d5df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:44.424856 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-4905d2e9-a30f-444c-8fc6-a41f68cfdce7 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:11:44.431559 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-4905d2e9-a30f-444c-8fc6-a41f68cfdce7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:11:44.444286 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-cf470719-9a0b-4c93-afad-664c1129d5df None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:11:44.446481 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-cf470719-9a0b-4c93-afad-664c1129d5df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:11:44.448636 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 779/1543] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:11:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 69 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:11:44.449720 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4905d2e9-a30f-444c-8fc6-a41f68cfdce7 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:44.470810 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4905d2e9-a30f-444c-8fc6-a41f68cfdce7 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:44.471228 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 765/1544] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:11:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:11:44.477031 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf57c90-de11-4695-876c-531c5c8f7308 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:44.477031 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf57c90-de11-4695-876c-531c5c8f7308 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:44.477031 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf57c90-de11-4695-876c-531c5c8f7308 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:44.485706 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d950e4d-f53c-4852-b273-8906218f03c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:44.486263 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d950e4d-f53c-4852-b273-8906218f03c3 admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:44.486541 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d950e4d-f53c-4852-b273-8906218f03c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:44.486810 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d950e4d-f53c-4852-b273-8906218f03c3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:44.487467 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d950e4d-f53c-4852-b273-8906218f03c3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:44.492830 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d950e4d-f53c-4852-b273-8906218f03c3 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:44.505228 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-3cf57c90-de11-4695-876c-531c5c8f7308 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:11:44.506436 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-3cf57c90-de11-4695-876c-531c5c8f7308 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:11:44.515145 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 766/1545] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:11:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 43 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:11:44.516270 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d2da757-c1f4-4e34-ba2d-23f52c44d78b None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:44.516556 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d2da757-c1f4-4e34-ba2d-23f52c44d78b None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:44.516825 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d2da757-c1f4-4e34-ba2d-23f52c44d78b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:44.539048 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-4d2da757-c1f4-4e34-ba2d-23f52c44d78b None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:11:44.540539 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-4d2da757-c1f4-4e34-ba2d-23f52c44d78b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:11:44.545841 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 780/1546] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:11:44 2022] POST /identity/v3/projects => generated 369 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:11:44.548008 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be526560-f92b-476b-adc6-bb230d317cf9 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:44.548580 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be526560-f92b-476b-adc6-bb230d317cf9 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:44.549099 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be526560-f92b-476b-adc6-bb230d317cf9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:44.549563 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 767/1547] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:11:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:11:44.551184 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08b46cd9-4c72-421b-b61c-e786d2c7b877 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:44.552058 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08b46cd9-4c72-421b-b61c-e786d2c7b877 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:44.552330 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08b46cd9-4c72-421b-b61c-e786d2c7b877 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:44.573331 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-08b46cd9-4c72-421b-b61c-e786d2c7b877 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:11:44.573785 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-be526560-f92b-476b-adc6-bb230d317cf9 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:11:44.574591 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-08b46cd9-4c72-421b-b61c-e786d2c7b877 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:11:44.575518 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-be526560-f92b-476b-adc6-bb230d317cf9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:11:44.639175 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 781/1548] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:11:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:11:44.640490 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c251c7bf-23e5-43cc-9fb3-55c1124d126b None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:44.640779 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c251c7bf-23e5-43cc-9fb3-55c1124d126b None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:44.641172 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c251c7bf-23e5-43cc-9fb3-55c1124d126b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:44.645776 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 768/1549] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:11:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:11:44.645776 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e35762-ea1c-4672-ae70-f8fdd79e2131 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:44.645776 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e35762-ea1c-4672-ae70-f8fdd79e2131 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:44.645776 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e35762-ea1c-4672-ae70-f8fdd79e2131 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:44.661060 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-c251c7bf-23e5-43cc-9fb3-55c1124d126b None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:11:44.662248 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-c251c7bf-23e5-43cc-9fb3-55c1124d126b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:11:44.663636 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-81e35762-ea1c-4672-ae70-f8fdd79e2131 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:11:44.664784 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-81e35762-ea1c-4672-ae70-f8fdd79e2131 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:11:44.670106 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 782/1550] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:11:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:11:44.670855 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c251c7bf-23e5-43cc-9fb3-55c1124d126b None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:44.679061 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 769/1551] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:11:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:11:44.680285 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7358a601-fb15-4709-90ac-382fbbb8d806 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:44.680572 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7358a601-fb15-4709-90ac-382fbbb8d806 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:44.680846 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7358a601-fb15-4709-90ac-382fbbb8d806 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:44.680940 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c251c7bf-23e5-43cc-9fb3-55c1124d126b None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:44.685812 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b5e631f-de4f-42c1-ab37-41365bd17ad2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:44.686441 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b5e631f-de4f-42c1-ab37-41365bd17ad2 admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:44.686716 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b5e631f-de4f-42c1-ab37-41365bd17ad2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:44.687016 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b5e631f-de4f-42c1-ab37-41365bd17ad2 admin admin] PATH_INFO: `/v3/users` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:44.687632 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b5e631f-de4f-42c1-ab37-41365bd17ad2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:44.688648 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: WARNING py.warnings [None req-0b5e631f-de4f-42c1-ab37-41365bd17ad2 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 30 05:11:44.688648 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: warnings.warn(msg) Aug 30 05:11:44.688648 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: Aug 30 05:11:44.689903 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b5e631f-de4f-42c1-ab37-41365bd17ad2 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:44.701442 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-7358a601-fb15-4709-90ac-382fbbb8d806 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:11:44.702973 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-7358a601-fb15-4709-90ac-382fbbb8d806 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:11:44.711384 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 783/1552] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:11:44 2022] POST /identity/v3/users => generated 285 bytes in 41 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:11:44.712314 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 770/1553] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:11:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:11:44.712847 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02d4a003-0463-4710-98cd-be3bf83c5e32 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:44.713216 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02d4a003-0463-4710-98cd-be3bf83c5e32 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:44.713330 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-767a38b2-cd44-48f3-9845-120a9e2ebd8b None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:44.713601 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02d4a003-0463-4710-98cd-be3bf83c5e32 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:44.713722 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-767a38b2-cd44-48f3-9845-120a9e2ebd8b None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:44.713866 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-767a38b2-cd44-48f3-9845-120a9e2ebd8b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:44.734003 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-767a38b2-cd44-48f3-9845-120a9e2ebd8b None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:11:44.735162 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-02d4a003-0463-4710-98cd-be3bf83c5e32 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:11:44.735320 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-767a38b2-cd44-48f3-9845-120a9e2ebd8b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:11:44.736398 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-02d4a003-0463-4710-98cd-be3bf83c5e32 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:11:44.743396 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 771/1554] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:11:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:11:44.744169 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-767a38b2-cd44-48f3-9845-120a9e2ebd8b None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:44.744591 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 784/1555] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:11:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:11:44.745317 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02d4a003-0463-4710-98cd-be3bf83c5e32 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:44.759793 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02d4a003-0463-4710-98cd-be3bf83c5e32 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:44.760212 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-767a38b2-cd44-48f3-9845-120a9e2ebd8b None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:44.764518 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2868e03-74be-4b30-a457-01126f07ba8b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:44.764904 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f304000b-9c8d-4625-b688-e16964bece7a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:44.765107 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2868e03-74be-4b30-a457-01126f07ba8b admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:44.765491 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f304000b-9c8d-4625-b688-e16964bece7a admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:44.765600 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2868e03-74be-4b30-a457-01126f07ba8b admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:44.765763 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2868e03-74be-4b30-a457-01126f07ba8b admin admin] PATH_INFO: `/v3/domains` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:44.765873 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f304000b-9c8d-4625-b688-e16964bece7a admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:44.766031 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f304000b-9c8d-4625-b688-e16964bece7a admin admin] PATH_INFO: `/v3/domains` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:44.766409 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2868e03-74be-4b30-a457-01126f07ba8b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:11:44.766669 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f304000b-9c8d-4625-b688-e16964bece7a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:11:44.766770 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2868e03-74be-4b30-a457-01126f07ba8b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:44.767006 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f304000b-9c8d-4625-b688-e16964bece7a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:44.767733 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: WARNING py.warnings [None req-f2868e03-74be-4b30-a457-01126f07ba8b admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Aug 30 05:11:44.767733 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: warnings.warn(msg) Aug 30 05:11:44.767733 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: Aug 30 05:11:44.768612 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2868e03-74be-4b30-a457-01126f07ba8b admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:44.768732 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f304000b-9c8d-4625-b688-e16964bece7a admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:44.780930 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 772/1556] 10.222.0.77 () {58 vars in 1153 bytes} [Tue Aug 30 05:11:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:11:44.782095 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eacaccca-9eda-4ec8-bac4-45b0e85311d2 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:44.782374 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eacaccca-9eda-4ec8-bac4-45b0e85311d2 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:44.782642 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eacaccca-9eda-4ec8-bac4-45b0e85311d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:44.783103 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 785/1557] 10.222.0.77 () {58 vars in 1153 bytes} [Tue Aug 30 05:11:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:11:44.784086 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2868e03-74be-4b30-a457-01126f07ba8b admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:44.794966 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2868e03-74be-4b30-a457-01126f07ba8b admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:44.800503 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41761bd9-38a1-4d85-8727-045d2a5027fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:44.801112 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41761bd9-38a1-4d85-8727-045d2a5027fa admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:44.801418 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41761bd9-38a1-4d85-8727-045d2a5027fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:44.801737 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41761bd9-38a1-4d85-8727-045d2a5027fa admin admin] PATH_INFO: `/v3/domains` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:44.802370 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41761bd9-38a1-4d85-8727-045d2a5027fa admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:11:44.802669 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41761bd9-38a1-4d85-8727-045d2a5027fa admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:44.804356 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41761bd9-38a1-4d85-8727-045d2a5027fa admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:44.806636 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-eacaccca-9eda-4ec8-bac4-45b0e85311d2 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:11:44.807877 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-eacaccca-9eda-4ec8-bac4-45b0e85311d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:11:44.816036 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 773/1558] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:11:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:11:44.816555 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 786/1559] 10.222.0.77 () {58 vars in 1153 bytes} [Tue Aug 30 05:11:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:11:44.816925 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eacaccca-9eda-4ec8-bac4-45b0e85311d2 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:44.817447 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41761bd9-38a1-4d85-8727-045d2a5027fa admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:44.826608 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eacaccca-9eda-4ec8-bac4-45b0e85311d2 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:44.831301 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df27dfa6-42d9-4433-9596-a6eff04dcdaa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:44.831587 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41761bd9-38a1-4d85-8727-045d2a5027fa admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:44.831896 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df27dfa6-42d9-4433-9596-a6eff04dcdaa admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:44.832177 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df27dfa6-42d9-4433-9596-a6eff04dcdaa admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:44.833008 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df27dfa6-42d9-4433-9596-a6eff04dcdaa admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:44.833646 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df27dfa6-42d9-4433-9596-a6eff04dcdaa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:44.835269 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df27dfa6-42d9-4433-9596-a6eff04dcdaa admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:44.836767 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec70d389-28ba-40de-8665-418e93b78714 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:44.837351 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec70d389-28ba-40de-8665-418e93b78714 admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:44.837630 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec70d389-28ba-40de-8665-418e93b78714 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:44.837898 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec70d389-28ba-40de-8665-418e93b78714 admin admin] PATH_INFO: `/v3/domains` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:44.838523 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec70d389-28ba-40de-8665-418e93b78714 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:11:44.838841 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec70d389-28ba-40de-8665-418e93b78714 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:44.840416 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec70d389-28ba-40de-8665-418e93b78714 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:44.846231 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 774/1560] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:11:44 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:11:44.847481 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3926a601-621f-4a9f-a77e-afed4e27194b None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:44.847765 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3926a601-621f-4a9f-a77e-afed4e27194b None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:44.848096 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3926a601-621f-4a9f-a77e-afed4e27194b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:44.854837 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 787/1561] 10.222.0.77 () {58 vars in 1153 bytes} [Tue Aug 30 05:11:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:11:44.856140 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e64c8392-6e5a-4e2d-a8fb-5597cf572ac1 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:44.856432 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e64c8392-6e5a-4e2d-a8fb-5597cf572ac1 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:44.856704 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e64c8392-6e5a-4e2d-a8fb-5597cf572ac1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:44.874087 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-3926a601-621f-4a9f-a77e-afed4e27194b None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:11:44.875240 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-e64c8392-6e5a-4e2d-a8fb-5597cf572ac1 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:11:44.875354 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-3926a601-621f-4a9f-a77e-afed4e27194b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:11:44.876405 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-e64c8392-6e5a-4e2d-a8fb-5597cf572ac1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:11:44.883129 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 775/1562] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:11:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:11:44.883866 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3926a601-621f-4a9f-a77e-afed4e27194b None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:44.886873 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 788/1563] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:11:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:11:44.886873 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e64c8392-6e5a-4e2d-a8fb-5597cf572ac1 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:44.896727 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e64c8392-6e5a-4e2d-a8fb-5597cf572ac1 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:44.896886 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3926a601-621f-4a9f-a77e-afed4e27194b None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:44.901216 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a501523-5cec-4b26-970a-70637cf87faa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:44.901432 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27cde35d-9979-44cb-84c7-5fabd45b9c9d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:44.901794 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a501523-5cec-4b26-970a-70637cf87faa admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:44.902019 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27cde35d-9979-44cb-84c7-5fabd45b9c9d admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:44.902116 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a501523-5cec-4b26-970a-70637cf87faa admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:44.902297 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27cde35d-9979-44cb-84c7-5fabd45b9c9d admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:44.902400 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a501523-5cec-4b26-970a-70637cf87faa admin admin] PATH_INFO: `/v3/projects` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:44.902570 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27cde35d-9979-44cb-84c7-5fabd45b9c9d admin admin] PATH_INFO: `/v3/projects` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:44.902959 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a501523-5cec-4b26-970a-70637cf87faa admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:44.903245 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27cde35d-9979-44cb-84c7-5fabd45b9c9d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:44.904248 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: WARNING py.warnings [None req-27cde35d-9979-44cb-84c7-5fabd45b9c9d admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 30 05:11:44.904248 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: warnings.warn(msg) Aug 30 05:11:44.904248 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: Aug 30 05:11:44.904874 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a501523-5cec-4b26-970a-70637cf87faa admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:44.905511 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27cde35d-9979-44cb-84c7-5fabd45b9c9d admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:44.932564 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 789/1564] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:11:44 2022] POST /identity/v3/projects => generated 377 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:11:44.933360 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a501523-5cec-4b26-970a-70637cf87faa admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:44.935861 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 776/1565] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:11:44 2022] POST /identity/v3/projects => generated 405 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:11:44.939628 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27cde35d-9979-44cb-84c7-5fabd45b9c9d admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:44.953518 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:11:44.987059 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:11:44.999603 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a501523-5cec-4b26-970a-70637cf87faa admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:45.004875 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b490041-172a-4f7e-b832-d8f5019315b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:45.005535 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b490041-172a-4f7e-b832-d8f5019315b0 admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:45.005828 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b490041-172a-4f7e-b832-d8f5019315b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:45.006099 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b490041-172a-4f7e-b832-d8f5019315b0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:45.006643 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27cde35d-9979-44cb-84c7-5fabd45b9c9d admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:45.006775 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b490041-172a-4f7e-b832-d8f5019315b0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:45.008665 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b490041-172a-4f7e-b832-d8f5019315b0 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:45.012411 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6290068b-85b9-4ca3-afdf-e55d79213343 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:45.013047 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6290068b-85b9-4ca3-afdf-e55d79213343 admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:45.013323 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6290068b-85b9-4ca3-afdf-e55d79213343 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:45.013591 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6290068b-85b9-4ca3-afdf-e55d79213343 admin admin] PATH_INFO: `/v3/domains` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:45.014230 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6290068b-85b9-4ca3-afdf-e55d79213343 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:11:45.014523 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6290068b-85b9-4ca3-afdf-e55d79213343 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:45.016168 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6290068b-85b9-4ca3-afdf-e55d79213343 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:45.029398 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 777/1566] 10.222.0.77 () {58 vars in 1153 bytes} [Tue Aug 30 05:11:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 90 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:11:45.030220 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6290068b-85b9-4ca3-afdf-e55d79213343 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:45.038083 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 790/1567] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:11:44 2022] POST /identity/v3/projects => generated 379 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:11:45.039038 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b490041-172a-4f7e-b832-d8f5019315b0 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:45.045082 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6290068b-85b9-4ca3-afdf-e55d79213343 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:45.101606 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d31856e-28fc-46bc-8fd2-977d983a7edc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:45.102009 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b490041-172a-4f7e-b832-d8f5019315b0 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:45.102211 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d31856e-28fc-46bc-8fd2-977d983a7edc admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:45.102491 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d31856e-28fc-46bc-8fd2-977d983a7edc admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:45.102765 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d31856e-28fc-46bc-8fd2-977d983a7edc admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:45.103426 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d31856e-28fc-46bc-8fd2-977d983a7edc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:45.105007 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d31856e-28fc-46bc-8fd2-977d983a7edc admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:45.106725 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3b92907-a0ba-4c4c-ad88-859a2f07791a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:45.107349 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3b92907-a0ba-4c4c-ad88-859a2f07791a admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:45.107625 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3b92907-a0ba-4c4c-ad88-859a2f07791a admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:45.107890 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3b92907-a0ba-4c4c-ad88-859a2f07791a admin admin] PATH_INFO: `/v3/projects` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:45.108510 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3b92907-a0ba-4c4c-ad88-859a2f07791a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:45.110396 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3b92907-a0ba-4c4c-ad88-859a2f07791a admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:45.114753 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 778/1568] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:11:45 2022] GET /identity/v3/roles => generated 1420 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:11:45.115709 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d31856e-28fc-46bc-8fd2-977d983a7edc admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:45.125354 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d31856e-28fc-46bc-8fd2-977d983a7edc admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:45.130604 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f313b6c7-2fb3-44c0-b696-6769184c07ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:45.131249 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f313b6c7-2fb3-44c0-b696-6769184c07ae admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:45.131530 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f313b6c7-2fb3-44c0-b696-6769184c07ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:45.131803 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f313b6c7-2fb3-44c0-b696-6769184c07ae admin admin] PATH_INFO: `/v3/domains` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:45.132531 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f313b6c7-2fb3-44c0-b696-6769184c07ae admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:11:45.132834 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f313b6c7-2fb3-44c0-b696-6769184c07ae admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:45.134508 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f313b6c7-2fb3-44c0-b696-6769184c07ae admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:45.144034 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 791/1569] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:11:45 2022] POST /identity/v3/projects => generated 393 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:11:45.144034 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3b92907-a0ba-4c4c-ad88-859a2f07791a admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:45.150067 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 779/1570] 10.222.0.77 () {58 vars in 1153 bytes} [Tue Aug 30 05:11:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:11:45.151004 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f313b6c7-2fb3-44c0-b696-6769184c07ae admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:45.166623 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f313b6c7-2fb3-44c0-b696-6769184c07ae admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:45.204317 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3b92907-a0ba-4c4c-ad88-859a2f07791a admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:45.209733 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eb21418-e5a6-4242-abf1-7ddbe5ef7923 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:45.210464 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eb21418-e5a6-4242-abf1-7ddbe5ef7923 admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:45.210746 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eb21418-e5a6-4242-abf1-7ddbe5ef7923 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:45.211040 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eb21418-e5a6-4242-abf1-7ddbe5ef7923 admin admin] PATH_INFO: `/v3/domains` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:45.211687 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eb21418-e5a6-4242-abf1-7ddbe5ef7923 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:11:45.211989 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eb21418-e5a6-4242-abf1-7ddbe5ef7923 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:45.213584 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eb21418-e5a6-4242-abf1-7ddbe5ef7923 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:45.234410 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 792/1571] 10.222.0.77 () {58 vars in 1153 bytes} [Tue Aug 30 05:11:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:11:45.234410 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eb21418-e5a6-4242-abf1-7ddbe5ef7923 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:45.234792 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92b04f93-c63f-4aa3-b928-a16d8c042531 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:45.234792 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b04f93-c63f-4aa3-b928-a16d8c042531 admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:45.234792 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b04f93-c63f-4aa3-b928-a16d8c042531 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:45.234792 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b04f93-c63f-4aa3-b928-a16d8c042531 admin admin] PATH_INFO: `/v3/users` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:45.235551 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92b04f93-c63f-4aa3-b928-a16d8c042531 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:45.236909 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92b04f93-c63f-4aa3-b928-a16d8c042531 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:45.241404 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eb21418-e5a6-4242-abf1-7ddbe5ef7923 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:45.246515 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fadc18d-87b8-483f-b569-172d0f761335 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:45.247142 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fadc18d-87b8-483f-b569-172d0f761335 admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:45.247455 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fadc18d-87b8-483f-b569-172d0f761335 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:45.247729 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fadc18d-87b8-483f-b569-172d0f761335 admin admin] PATH_INFO: `/v3/users` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:45.248359 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fadc18d-87b8-483f-b569-172d0f761335 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:45.251016 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fadc18d-87b8-483f-b569-172d0f761335 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:45.261960 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 780/1572] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:11:45 2022] POST /identity/v3/users => generated 289 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:11:45.262937 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92b04f93-c63f-4aa3-b928-a16d8c042531 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:45.274150 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92b04f93-c63f-4aa3-b928-a16d8c042531 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:45.274339 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 793/1573] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:11:45 2022] POST /identity/v3/users => generated 303 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:11:45.277817 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fadc18d-87b8-483f-b569-172d0f761335 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:45.279253 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-878e7276-ecac-45c4-8ac4-4ccc4e28f7f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:45.279948 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-878e7276-ecac-45c4-8ac4-4ccc4e28f7f3 admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:45.280323 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-878e7276-ecac-45c4-8ac4-4ccc4e28f7f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:45.280593 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-878e7276-ecac-45c4-8ac4-4ccc4e28f7f3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:45.281214 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-878e7276-ecac-45c4-8ac4-4ccc4e28f7f3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:45.283201 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-878e7276-ecac-45c4-8ac4-4ccc4e28f7f3 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:45.283834 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:11:45.288446 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fadc18d-87b8-483f-b569-172d0f761335 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:45.294165 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef1e08f5-bda7-42f9-8ac9-d5c2958072ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:45.294755 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef1e08f5-bda7-42f9-8ac9-d5c2958072ea admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:45.295074 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef1e08f5-bda7-42f9-8ac9-d5c2958072ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:45.295348 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef1e08f5-bda7-42f9-8ac9-d5c2958072ea admin admin] PATH_INFO: `/v3/users` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:45.295971 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef1e08f5-bda7-42f9-8ac9-d5c2958072ea admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:45.297882 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef1e08f5-bda7-42f9-8ac9-d5c2958072ea admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:45.312812 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 781/1574] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:11:45 2022] POST /identity/v3/projects => generated 387 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:11:45.313602 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-878e7276-ecac-45c4-8ac4-4ccc4e28f7f3 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:45.321722 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 794/1575] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:11:45 2022] POST /identity/v3/users => generated 290 bytes in 44 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:11:45.322641 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef1e08f5-bda7-42f9-8ac9-d5c2958072ea admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:45.332153 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef1e08f5-bda7-42f9-8ac9-d5c2958072ea admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:45.346221 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:11:45.378589 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-878e7276-ecac-45c4-8ac4-4ccc4e28f7f3 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:45.383737 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa551aaf-3f62-4117-949a-60e662b69603 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:45.384387 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa551aaf-3f62-4117-949a-60e662b69603 admin admin] REQUEST_METHOD: `PUT` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:45.384666 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa551aaf-3f62-4117-949a-60e662b69603 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:45.384934 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa551aaf-3f62-4117-949a-60e662b69603 admin admin] PATH_INFO: `/v3/projects/b8c7cffa472b424eb9052b47f2bbce8c/users/f17b1949106d4fd1b0af8bd13b059c6a/roles/6f03d86980c34c36b44a8d6764ff4fda` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:45.390686 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34b8fa4a-afc3-4ed5-96af-329e415521ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:45.391313 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34b8fa4a-afc3-4ed5-96af-329e415521ce admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:45.391591 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34b8fa4a-afc3-4ed5-96af-329e415521ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:45.391861 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34b8fa4a-afc3-4ed5-96af-329e415521ce admin admin] PATH_INFO: `/v3/users` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:45.392482 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34b8fa4a-afc3-4ed5-96af-329e415521ce admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:45.394392 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34b8fa4a-afc3-4ed5-96af-329e415521ce admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:45.399566 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa551aaf-3f62-4117-949a-60e662b69603 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b8c7cffa472b424eb9052b47f2bbce8c, user_id=f17b1949106d4fd1b0af8bd13b059c6a, role_id=6f03d86980c34c36b44a8d6764ff4fda)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:45.400593 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: WARNING py.warnings [None req-aa551aaf-3f62-4117-949a-60e662b69603 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 30 05:11:45.400593 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: warnings.warn(msg) Aug 30 05:11:45.400593 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: Aug 30 05:11:45.402502 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa551aaf-3f62-4117-949a-60e662b69603 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:45.415894 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 782/1576] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:11:45 2022] PUT /identity/v3/projects/b8c7cffa472b424eb9052b47f2bbce8c/users/f17b1949106d4fd1b0af8bd13b059c6a/roles/6f03d86980c34c36b44a8d6764ff4fda => generated 0 bytes in 103 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:11:45.419459 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 795/1577] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:11:45 2022] POST /identity/v3/users => generated 297 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:11:45.419459 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34b8fa4a-afc3-4ed5-96af-329e415521ce admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:45.419694 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa551aaf-3f62-4117-949a-60e662b69603 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:45.427929 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34b8fa4a-afc3-4ed5-96af-329e415521ce admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:45.486676 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa551aaf-3f62-4117-949a-60e662b69603 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:45.487879 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04c0be66-ddc2-4fc8-8b8b-d5afe15c5a70 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:45.488479 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04c0be66-ddc2-4fc8-8b8b-d5afe15c5a70 admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:45.489272 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04c0be66-ddc2-4fc8-8b8b-d5afe15c5a70 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:45.489535 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04c0be66-ddc2-4fc8-8b8b-d5afe15c5a70 admin admin] PATH_INFO: `/v3/projects` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:45.490151 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04c0be66-ddc2-4fc8-8b8b-d5afe15c5a70 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:45.492110 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82e3c84f-839d-47db-9f30-335407c3e4dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:45.492234 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04c0be66-ddc2-4fc8-8b8b-d5afe15c5a70 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:45.492657 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82e3c84f-839d-47db-9f30-335407c3e4dc admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:45.492934 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82e3c84f-839d-47db-9f30-335407c3e4dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:45.493203 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82e3c84f-839d-47db-9f30-335407c3e4dc admin admin] PATH_INFO: `/v3/projects` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:45.493820 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82e3c84f-839d-47db-9f30-335407c3e4dc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:45.495742 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82e3c84f-839d-47db-9f30-335407c3e4dc admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:45.519031 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 796/1578] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:11:45 2022] POST /identity/v3/projects => generated 363 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:11:45.519861 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04c0be66-ddc2-4fc8-8b8b-d5afe15c5a70 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:45.522734 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 783/1579] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:11:45 2022] POST /identity/v3/projects => generated 367 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:11:45.523588 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82e3c84f-839d-47db-9f30-335407c3e4dc admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:45.583050 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04c0be66-ddc2-4fc8-8b8b-d5afe15c5a70 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:45.584500 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82e3c84f-839d-47db-9f30-335407c3e4dc admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:45.587709 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4826f433-4911-4c5a-90b9-b139d641a3f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:45.588299 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4826f433-4911-4c5a-90b9-b139d641a3f4 admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:45.588574 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4826f433-4911-4c5a-90b9-b139d641a3f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:45.588838 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4826f433-4911-4c5a-90b9-b139d641a3f4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:45.589033 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23b68130-17b4-4bba-a13b-09f746100bd6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:45.589468 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4826f433-4911-4c5a-90b9-b139d641a3f4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:45.589621 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b68130-17b4-4bba-a13b-09f746100bd6 admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:45.589896 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b68130-17b4-4bba-a13b-09f746100bd6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:45.590162 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b68130-17b4-4bba-a13b-09f746100bd6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:45.590785 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23b68130-17b4-4bba-a13b-09f746100bd6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:45.591076 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4826f433-4911-4c5a-90b9-b139d641a3f4 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:45.592686 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23b68130-17b4-4bba-a13b-09f746100bd6 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:45.593076 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:11:45.602998 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 797/1580] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:11:45 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:11:45.604016 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4826f433-4911-4c5a-90b9-b139d641a3f4 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:45.604145 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 784/1581] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:11:45 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:11:45.605152 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23b68130-17b4-4bba-a13b-09f746100bd6 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:45.615058 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4826f433-4911-4c5a-90b9-b139d641a3f4 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:45.615529 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23b68130-17b4-4bba-a13b-09f746100bd6 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:45.619680 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08c2a65f-b782-45ad-adcd-ee7a31e1b14b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:45.620330 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08c2a65f-b782-45ad-adcd-ee7a31e1b14b admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:45.620498 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c91d7b56-b854-4c2f-bf09-d0ce738b95bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:45.620609 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08c2a65f-b782-45ad-adcd-ee7a31e1b14b admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:45.620876 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08c2a65f-b782-45ad-adcd-ee7a31e1b14b admin admin] PATH_INFO: `/v3/users` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:45.621360 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c91d7b56-b854-4c2f-bf09-d0ce738b95bc admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:45.621616 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08c2a65f-b782-45ad-adcd-ee7a31e1b14b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:45.621925 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c91d7b56-b854-4c2f-bf09-d0ce738b95bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:45.622206 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c91d7b56-b854-4c2f-bf09-d0ce738b95bc admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:45.622869 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c91d7b56-b854-4c2f-bf09-d0ce738b95bc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:45.623449 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08c2a65f-b782-45ad-adcd-ee7a31e1b14b admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:45.628838 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c91d7b56-b854-4c2f-bf09-d0ce738b95bc admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:45.641384 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 785/1582] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:11:45 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:11:45.642605 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c31d6c61-b02d-49ee-8c27-16c3d6abe232 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:45.642937 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c31d6c61-b02d-49ee-8c27-16c3d6abe232 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:45.643206 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c31d6c61-b02d-49ee-8c27-16c3d6abe232 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:45.646650 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 798/1583] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:11:45 2022] POST /identity/v3/users => generated 294 bytes in 43 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:11:45.647523 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08c2a65f-b782-45ad-adcd-ee7a31e1b14b admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:45.657094 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08c2a65f-b782-45ad-adcd-ee7a31e1b14b admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:45.661689 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1119174e-14fa-42b2-8b27-a47d071fd7d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:45.662261 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1119174e-14fa-42b2-8b27-a47d071fd7d3 admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:45.662538 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1119174e-14fa-42b2-8b27-a47d071fd7d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:45.662805 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1119174e-14fa-42b2-8b27-a47d071fd7d3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:45.663455 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1119174e-14fa-42b2-8b27-a47d071fd7d3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:45.665065 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1119174e-14fa-42b2-8b27-a47d071fd7d3 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:45.673882 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-c31d6c61-b02d-49ee-8c27-16c3d6abe232 None None] MFA Rules not processed for user `f17b1949106d4fd1b0af8bd13b059c6a`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:11:45.675111 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-c31d6c61-b02d-49ee-8c27-16c3d6abe232 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:11:45.675340 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 799/1584] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:11:45 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:11:45.676104 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1119174e-14fa-42b2-8b27-a47d071fd7d3 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:45.683802 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:11:45.686048 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1119174e-14fa-42b2-8b27-a47d071fd7d3 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:45.690920 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1e6cd54-af1f-4012-81bd-87bf553d848d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:45.691501 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e6cd54-af1f-4012-81bd-87bf553d848d admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:45.691798 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e6cd54-af1f-4012-81bd-87bf553d848d admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:45.692596 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e6cd54-af1f-4012-81bd-87bf553d848d admin admin] PATH_INFO: `/v3/users` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:45.693212 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1e6cd54-af1f-4012-81bd-87bf553d848d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:45.695116 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1e6cd54-af1f-4012-81bd-87bf553d848d admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:45.717967 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 800/1585] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:11:45 2022] POST /identity/v3/users => generated 282 bytes in 42 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:11:45.719066 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1e6cd54-af1f-4012-81bd-87bf553d848d admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:45.728956 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1e6cd54-af1f-4012-81bd-87bf553d848d admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:45.734049 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17dac9c3-cc96-4315-afee-f6ae3a7d7fab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:45.734623 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17dac9c3-cc96-4315-afee-f6ae3a7d7fab admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:45.734929 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17dac9c3-cc96-4315-afee-f6ae3a7d7fab admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:45.735203 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17dac9c3-cc96-4315-afee-f6ae3a7d7fab admin admin] PATH_INFO: `/v3/users` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:45.735817 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17dac9c3-cc96-4315-afee-f6ae3a7d7fab admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:45.737823 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17dac9c3-cc96-4315-afee-f6ae3a7d7fab admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:45.741795 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 786/1586] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:11:45 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:11:45.746019 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c31d6c61-b02d-49ee-8c27-16c3d6abe232 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:45.758060 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c31d6c61-b02d-49ee-8c27-16c3d6abe232 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:45.759485 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 801/1587] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:11:45 2022] POST /identity/v3/users => generated 284 bytes in 41 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:11:45.760284 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17dac9c3-cc96-4315-afee-f6ae3a7d7fab admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:45.765788 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94bcaf56-a27a-4ee5-990b-51df45b9672f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:45.766406 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94bcaf56-a27a-4ee5-990b-51df45b9672f admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:45.766680 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94bcaf56-a27a-4ee5-990b-51df45b9672f admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:45.766982 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94bcaf56-a27a-4ee5-990b-51df45b9672f admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:45.767608 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94bcaf56-a27a-4ee5-990b-51df45b9672f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:45.769198 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94bcaf56-a27a-4ee5-990b-51df45b9672f admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:45.769772 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17dac9c3-cc96-4315-afee-f6ae3a7d7fab admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:45.775048 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ee85592-03a1-4ff7-a8dd-97d0866e956f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:45.775744 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee85592-03a1-4ff7-a8dd-97d0866e956f admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:45.776030 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee85592-03a1-4ff7-a8dd-97d0866e956f admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:45.776460 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee85592-03a1-4ff7-a8dd-97d0866e956f admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:45.777090 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ee85592-03a1-4ff7-a8dd-97d0866e956f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:45.778655 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ee85592-03a1-4ff7-a8dd-97d0866e956f admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:45.782357 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 787/1588] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:11:45 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:11:45.782357 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94bcaf56-a27a-4ee5-990b-51df45b9672f admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:45.789464 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94bcaf56-a27a-4ee5-990b-51df45b9672f admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:45.789873 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 802/1589] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:11:45 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:11:45.790544 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ee85592-03a1-4ff7-a8dd-97d0866e956f admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:45.797021 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ee543fe-b757-4d02-aebd-eeef51131aa8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:45.800363 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee543fe-b757-4d02-aebd-eeef51131aa8 admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:45.800363 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee543fe-b757-4d02-aebd-eeef51131aa8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:45.800363 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee543fe-b757-4d02-aebd-eeef51131aa8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:45.800363 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ee543fe-b757-4d02-aebd-eeef51131aa8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:45.800363 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ee543fe-b757-4d02-aebd-eeef51131aa8 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:45.802275 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ee85592-03a1-4ff7-a8dd-97d0866e956f admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:45.807056 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42d3540f-94a5-465a-8d89-5596955f746d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:45.807639 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42d3540f-94a5-465a-8d89-5596955f746d admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:45.807920 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42d3540f-94a5-465a-8d89-5596955f746d admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:45.808194 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42d3540f-94a5-465a-8d89-5596955f746d admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:45.808820 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42d3540f-94a5-465a-8d89-5596955f746d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:45.810417 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42d3540f-94a5-465a-8d89-5596955f746d admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:45.814092 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 788/1590] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:11:45 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:11:45.814092 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ee543fe-b757-4d02-aebd-eeef51131aa8 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:45.821798 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ee543fe-b757-4d02-aebd-eeef51131aa8 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:45.825024 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 803/1591] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:11:45 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:11:45.825024 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42d3540f-94a5-465a-8d89-5596955f746d admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:45.828828 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05aa1191-5d22-45f5-b5bf-45591704247b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:45.829415 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05aa1191-5d22-45f5-b5bf-45591704247b admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:45.829725 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05aa1191-5d22-45f5-b5bf-45591704247b admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:45.829986 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05aa1191-5d22-45f5-b5bf-45591704247b admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:45.830590 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05aa1191-5d22-45f5-b5bf-45591704247b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:45.832273 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05aa1191-5d22-45f5-b5bf-45591704247b admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:45.834053 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42d3540f-94a5-465a-8d89-5596955f746d admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:45.838606 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7d936af-ff92-4936-8162-757ed691b3c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:45.839212 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7d936af-ff92-4936-8162-757ed691b3c6 admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:45.839490 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7d936af-ff92-4936-8162-757ed691b3c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:45.839767 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7d936af-ff92-4936-8162-757ed691b3c6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:45.840381 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7d936af-ff92-4936-8162-757ed691b3c6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:45.841934 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7d936af-ff92-4936-8162-757ed691b3c6 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:45.843165 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 789/1592] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:11:45 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:11:45.844050 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05aa1191-5d22-45f5-b5bf-45591704247b admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:45.853316 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 804/1593] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:11:45 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:11:45.854510 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7d936af-ff92-4936-8162-757ed691b3c6 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:45.855096 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05aa1191-5d22-45f5-b5bf-45591704247b admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:45.859891 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0793b352-c917-41bf-bb3a-591e18979344 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:45.860525 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0793b352-c917-41bf-bb3a-591e18979344 admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:45.860803 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0793b352-c917-41bf-bb3a-591e18979344 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:45.861079 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0793b352-c917-41bf-bb3a-591e18979344 admin admin] PATH_INFO: `/v3/projects` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:45.861728 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0793b352-c917-41bf-bb3a-591e18979344 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:45.864400 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0793b352-c917-41bf-bb3a-591e18979344 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:45.865635 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7d936af-ff92-4936-8162-757ed691b3c6 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:45.870165 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7c2c663-7c64-4a8e-910b-22d68554a109 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:45.870735 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7c2c663-7c64-4a8e-910b-22d68554a109 admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:45.871043 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7c2c663-7c64-4a8e-910b-22d68554a109 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:45.871310 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7c2c663-7c64-4a8e-910b-22d68554a109 admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:45.871931 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7c2c663-7c64-4a8e-910b-22d68554a109 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:45.873473 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7c2c663-7c64-4a8e-910b-22d68554a109 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:45.883954 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 805/1594] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:11:45 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:11:45.884814 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7c2c663-7c64-4a8e-910b-22d68554a109 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:45.891565 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 790/1595] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:11:45 2022] POST /identity/v3/projects => generated 371 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:11:45.892510 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0793b352-c917-41bf-bb3a-591e18979344 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:45.896036 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7c2c663-7c64-4a8e-910b-22d68554a109 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:45.953117 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3d3f8b1-4380-4379-bdee-6d14e5dd6673 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:45.953768 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3d3f8b1-4380-4379-bdee-6d14e5dd6673 admin admin] REQUEST_METHOD: `PUT` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:45.954051 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3d3f8b1-4380-4379-bdee-6d14e5dd6673 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:45.954323 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3d3f8b1-4380-4379-bdee-6d14e5dd6673 admin admin] PATH_INFO: `/v3/projects/116fb506b7794c1fa9460b5a861de136/users/a8f01d0042c84767899dc2b71bca4e12/roles/6f03d86980c34c36b44a8d6764ff4fda` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:45.955009 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0793b352-c917-41bf-bb3a-591e18979344 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:45.960412 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd2b0b86-7ae9-4a0d-9807-b7abb4f941d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:45.961067 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd2b0b86-7ae9-4a0d-9807-b7abb4f941d8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:45.961350 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd2b0b86-7ae9-4a0d-9807-b7abb4f941d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:45.961627 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd2b0b86-7ae9-4a0d-9807-b7abb4f941d8 admin admin] PATH_INFO: `/v3/projects/30239b4a7a034b13b01b8572a2ec40cb/users/e6c0a8421dcd402cbc46505e4b292cce/roles/6f03d86980c34c36b44a8d6764ff4fda` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:45.969061 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3d3f8b1-4380-4379-bdee-6d14e5dd6673 admin admin] RBAC: Authorizing `identity:create_grant(project_id=116fb506b7794c1fa9460b5a861de136, user_id=a8f01d0042c84767899dc2b71bca4e12, role_id=6f03d86980c34c36b44a8d6764ff4fda)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:45.972056 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3d3f8b1-4380-4379-bdee-6d14e5dd6673 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:45.977947 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd2b0b86-7ae9-4a0d-9807-b7abb4f941d8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=30239b4a7a034b13b01b8572a2ec40cb, user_id=e6c0a8421dcd402cbc46505e4b292cce, role_id=6f03d86980c34c36b44a8d6764ff4fda)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:45.980725 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd2b0b86-7ae9-4a0d-9807-b7abb4f941d8 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:45.985985 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 806/1596] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:11:45 2022] PUT /identity/v3/projects/116fb506b7794c1fa9460b5a861de136/users/a8f01d0042c84767899dc2b71bca4e12/roles/6f03d86980c34c36b44a8d6764ff4fda => generated 0 bytes in 101 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:11:45.987088 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3d3f8b1-4380-4379-bdee-6d14e5dd6673 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:45.997248 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 791/1597] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:11:45 2022] PUT /identity/v3/projects/30239b4a7a034b13b01b8572a2ec40cb/users/e6c0a8421dcd402cbc46505e4b292cce/roles/6f03d86980c34c36b44a8d6764ff4fda => generated 0 bytes in 105 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:11:45.997982 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd2b0b86-7ae9-4a0d-9807-b7abb4f941d8 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:46.024475 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=90,keystone:INSERT=10 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:11:46.038353 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:11:46.054170 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3d3f8b1-4380-4379-bdee-6d14e5dd6673 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:46.058776 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc0f455d-551b-4fb5-9f13-74cb9cad0120 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:46.059452 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc0f455d-551b-4fb5-9f13-74cb9cad0120 admin admin] REQUEST_METHOD: `PUT` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:46.059727 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc0f455d-551b-4fb5-9f13-74cb9cad0120 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:46.059997 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc0f455d-551b-4fb5-9f13-74cb9cad0120 admin admin] PATH_INFO: `/v3/projects/0ed43db2f368485990e3add370edae5f/users/d611a49b03554b448a09fe86604af02b/roles/6f03d86980c34c36b44a8d6764ff4fda` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:46.061918 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd2b0b86-7ae9-4a0d-9807-b7abb4f941d8 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:46.066497 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3808a065-5959-4260-b3c8-4db0d35f2d93 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:46.067152 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3808a065-5959-4260-b3c8-4db0d35f2d93 admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:46.067444 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3808a065-5959-4260-b3c8-4db0d35f2d93 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:46.067691 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3808a065-5959-4260-b3c8-4db0d35f2d93 admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:46.068326 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3808a065-5959-4260-b3c8-4db0d35f2d93 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:46.069906 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3808a065-5959-4260-b3c8-4db0d35f2d93 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:46.074759 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc0f455d-551b-4fb5-9f13-74cb9cad0120 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0ed43db2f368485990e3add370edae5f, user_id=d611a49b03554b448a09fe86604af02b, role_id=6f03d86980c34c36b44a8d6764ff4fda)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:46.077444 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc0f455d-551b-4fb5-9f13-74cb9cad0120 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:46.079950 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 792/1598] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:11:45 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:11:46.082027 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3808a065-5959-4260-b3c8-4db0d35f2d93 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:46.091180 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 807/1599] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:11:45 2022] PUT /identity/v3/projects/0ed43db2f368485990e3add370edae5f/users/d611a49b03554b448a09fe86604af02b/roles/6f03d86980c34c36b44a8d6764ff4fda => generated 0 bytes in 105 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:11:46.091995 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3808a065-5959-4260-b3c8-4db0d35f2d93 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:46.092261 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc0f455d-551b-4fb5-9f13-74cb9cad0120 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:46.150703 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38a5e435-c79c-4a2e-b838-fe9268543037 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:46.151430 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38a5e435-c79c-4a2e-b838-fe9268543037 admin admin] REQUEST_METHOD: `PUT` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:46.151728 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38a5e435-c79c-4a2e-b838-fe9268543037 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:46.151987 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38a5e435-c79c-4a2e-b838-fe9268543037 admin admin] PATH_INFO: `/v3/projects/d2ce224c162c414c9be681b76801969c/users/31cc6aa564be4c569e6fa89813cb9329/roles/6f03d86980c34c36b44a8d6764ff4fda` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:46.156012 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc0f455d-551b-4fb5-9f13-74cb9cad0120 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:46.161143 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d958a573-079c-49c1-8a7b-ea15cb8f276d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:46.161744 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d958a573-079c-49c1-8a7b-ea15cb8f276d admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:46.162141 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d958a573-079c-49c1-8a7b-ea15cb8f276d admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:46.162417 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d958a573-079c-49c1-8a7b-ea15cb8f276d admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:46.163077 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d958a573-079c-49c1-8a7b-ea15cb8f276d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:46.164671 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d958a573-079c-49c1-8a7b-ea15cb8f276d admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:46.167727 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38a5e435-c79c-4a2e-b838-fe9268543037 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d2ce224c162c414c9be681b76801969c, user_id=31cc6aa564be4c569e6fa89813cb9329, role_id=6f03d86980c34c36b44a8d6764ff4fda)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:46.170900 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38a5e435-c79c-4a2e-b838-fe9268543037 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:46.174856 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 808/1600] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:11:46 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:11:46.175925 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d958a573-079c-49c1-8a7b-ea15cb8f276d admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:46.185948 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 793/1601] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:11:46 2022] PUT /identity/v3/projects/d2ce224c162c414c9be681b76801969c/users/31cc6aa564be4c569e6fa89813cb9329/roles/6f03d86980c34c36b44a8d6764ff4fda => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:11:46.186975 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38a5e435-c79c-4a2e-b838-fe9268543037 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:46.191889 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d958a573-079c-49c1-8a7b-ea15cb8f276d admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:46.248287 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38a5e435-c79c-4a2e-b838-fe9268543037 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:46.248823 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64e37797-3752-4a77-950b-4bcd632115fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:46.249118 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64e37797-3752-4a77-950b-4bcd632115fd admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:46.249400 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64e37797-3752-4a77-950b-4bcd632115fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:46.249671 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64e37797-3752-4a77-950b-4bcd632115fd admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:46.250304 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64e37797-3752-4a77-950b-4bcd632115fd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:46.251938 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64e37797-3752-4a77-950b-4bcd632115fd admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:46.252931 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eccbf299-3aaf-4182-bcc9-edbdfac2e278 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:46.253643 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eccbf299-3aaf-4182-bcc9-edbdfac2e278 admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:46.253926 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eccbf299-3aaf-4182-bcc9-edbdfac2e278 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:46.254204 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eccbf299-3aaf-4182-bcc9-edbdfac2e278 admin admin] PATH_INFO: `/v3/users` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:46.254839 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eccbf299-3aaf-4182-bcc9-edbdfac2e278 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:46.256822 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eccbf299-3aaf-4182-bcc9-edbdfac2e278 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:46.262600 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 809/1602] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:11:46 2022] GET /identity/v3/roles => generated 1420 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:11:46.263912 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c850f764-dab2-40c4-8cc9-f2ac40a0b610 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:46.264204 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c850f764-dab2-40c4-8cc9-f2ac40a0b610 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:46.264473 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c850f764-dab2-40c4-8cc9-f2ac40a0b610 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:46.281772 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 794/1603] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:11:46 2022] POST /identity/v3/users => generated 292 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:11:46.281772 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be24fd0-cf00-4d06-8ae9-5bbc0615546d None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:46.281772 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be24fd0-cf00-4d06-8ae9-5bbc0615546d None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:46.281772 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be24fd0-cf00-4d06-8ae9-5bbc0615546d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:46.284468 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-c850f764-dab2-40c4-8cc9-f2ac40a0b610 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:11:46.286432 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-c850f764-dab2-40c4-8cc9-f2ac40a0b610 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:11:46.294168 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 810/1604] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:11:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:11:46.294168 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c850f764-dab2-40c4-8cc9-f2ac40a0b610 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:46.303343 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-6be24fd0-cf00-4d06-8ae9-5bbc0615546d None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:11:46.304354 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c850f764-dab2-40c4-8cc9-f2ac40a0b610 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:46.304586 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-6be24fd0-cf00-4d06-8ae9-5bbc0615546d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:11:46.308897 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4b5f448-0e77-40e2-823d-d3b01751c283 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:46.309537 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4b5f448-0e77-40e2-823d-d3b01751c283 admin admin] REQUEST_METHOD: `PUT` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:46.309804 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4b5f448-0e77-40e2-823d-d3b01751c283 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:46.310072 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4b5f448-0e77-40e2-823d-d3b01751c283 admin admin] PATH_INFO: `/v3/projects/7ea4da0dbf3a44eabbff29f884beb157/users/2fe451987a8c49c5b5fc0ef1c9c30ab5/roles/6f03d86980c34c36b44a8d6764ff4fda` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:46.315207 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 795/1605] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:11:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:11:46.315207 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46ac7342-f2b7-4c8f-afbf-1d78ed3a4339 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:46.315207 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46ac7342-f2b7-4c8f-afbf-1d78ed3a4339 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:46.315207 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46ac7342-f2b7-4c8f-afbf-1d78ed3a4339 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:46.325338 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4b5f448-0e77-40e2-823d-d3b01751c283 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7ea4da0dbf3a44eabbff29f884beb157, user_id=2fe451987a8c49c5b5fc0ef1c9c30ab5, role_id=6f03d86980c34c36b44a8d6764ff4fda)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:46.328024 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4b5f448-0e77-40e2-823d-d3b01751c283 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:46.332125 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-46ac7342-f2b7-4c8f-afbf-1d78ed3a4339 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:11:46.333979 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-46ac7342-f2b7-4c8f-afbf-1d78ed3a4339 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:11:46.341354 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 811/1606] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:11:46 2022] PUT /identity/v3/projects/7ea4da0dbf3a44eabbff29f884beb157/users/2fe451987a8c49c5b5fc0ef1c9c30ab5/roles/6f03d86980c34c36b44a8d6764ff4fda => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:11:46.342298 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4b5f448-0e77-40e2-823d-d3b01751c283 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:46.362120 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:11:46.371280 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:11:46.398026 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 796/1607] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:11:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:11:46.399353 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51060cb3-d8a1-4f71-8ac4-70eafbae8a32 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:46.399637 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51060cb3-d8a1-4f71-8ac4-70eafbae8a32 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:46.399912 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51060cb3-d8a1-4f71-8ac4-70eafbae8a32 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:46.408745 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4b5f448-0e77-40e2-823d-d3b01751c283 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:46.414020 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b35ecf0-80ff-4855-97f4-3b2f4f5e8654 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:46.414665 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b35ecf0-80ff-4855-97f4-3b2f4f5e8654 admin admin] REQUEST_METHOD: `PUT` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:46.414980 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b35ecf0-80ff-4855-97f4-3b2f4f5e8654 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:46.415255 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b35ecf0-80ff-4855-97f4-3b2f4f5e8654 admin admin] PATH_INFO: `/v3/projects/558c2bea073546cabaffbf54e1d85a14/users/34204351339944f1995ce340ca36409b/roles/6f03d86980c34c36b44a8d6764ff4fda` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:46.418972 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-51060cb3-d8a1-4f71-8ac4-70eafbae8a32 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:11:46.420180 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-51060cb3-d8a1-4f71-8ac4-70eafbae8a32 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:11:46.430899 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 797/1608] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:11:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:11:46.430899 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51060cb3-d8a1-4f71-8ac4-70eafbae8a32 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:46.432688 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b35ecf0-80ff-4855-97f4-3b2f4f5e8654 admin admin] RBAC: Authorizing `identity:create_grant(project_id=558c2bea073546cabaffbf54e1d85a14, user_id=34204351339944f1995ce340ca36409b, role_id=6f03d86980c34c36b44a8d6764ff4fda)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:46.435564 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b35ecf0-80ff-4855-97f4-3b2f4f5e8654 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:46.438340 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51060cb3-d8a1-4f71-8ac4-70eafbae8a32 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:46.443678 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5bcb827-213c-41a5-9a45-9b54a9137373 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:46.444325 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5bcb827-213c-41a5-9a45-9b54a9137373 admin admin] REQUEST_METHOD: `PUT` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:46.444603 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5bcb827-213c-41a5-9a45-9b54a9137373 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:46.444871 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5bcb827-213c-41a5-9a45-9b54a9137373 admin admin] PATH_INFO: `/v3/projects/27a07b7f74cc48e29883a106e15996a8/users/5f1d61f4a3a84cbbaf9fe33f3225c58e/roles/6f03d86980c34c36b44a8d6764ff4fda` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:46.448631 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 812/1609] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:11:46 2022] PUT /identity/v3/projects/558c2bea073546cabaffbf54e1d85a14/users/34204351339944f1995ce340ca36409b/roles/6f03d86980c34c36b44a8d6764ff4fda => generated 0 bytes in 106 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:11:46.449213 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b35ecf0-80ff-4855-97f4-3b2f4f5e8654 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:46.465198 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5bcb827-213c-41a5-9a45-9b54a9137373 admin admin] RBAC: Authorizing `identity:create_grant(project_id=27a07b7f74cc48e29883a106e15996a8, user_id=5f1d61f4a3a84cbbaf9fe33f3225c58e, role_id=6f03d86980c34c36b44a8d6764ff4fda)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:46.468198 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5bcb827-213c-41a5-9a45-9b54a9137373 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:46.483095 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 798/1610] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:11:46 2022] PUT /identity/v3/projects/27a07b7f74cc48e29883a106e15996a8/users/5f1d61f4a3a84cbbaf9fe33f3225c58e/roles/6f03d86980c34c36b44a8d6764ff4fda => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:11:46.483970 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5bcb827-213c-41a5-9a45-9b54a9137373 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:46.524229 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b35ecf0-80ff-4855-97f4-3b2f4f5e8654 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:46.538751 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5bcb827-213c-41a5-9a45-9b54a9137373 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:46.542570 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-178b3233-cd8f-4912-aada-54ba73cf474e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:46.543218 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-178b3233-cd8f-4912-aada-54ba73cf474e service neutron] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:46.543500 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-178b3233-cd8f-4912-aada-54ba73cf474e service neutron] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:46.543771 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-178b3233-cd8f-4912-aada-54ba73cf474e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:46.581152 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93b23b59-4676-4148-a976-82796fecdcd3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:46.581752 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b23b59-4676-4148-a976-82796fecdcd3 admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:46.582025 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b23b59-4676-4148-a976-82796fecdcd3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:46.582293 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b23b59-4676-4148-a976-82796fecdcd3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:46.583494 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93b23b59-4676-4148-a976-82796fecdcd3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:46.585130 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93b23b59-4676-4148-a976-82796fecdcd3 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:46.594701 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 813/1611] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:11:46 2022] GET /identity/v3/roles => generated 1420 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:11:46.595772 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93b23b59-4676-4148-a976-82796fecdcd3 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:46.597751 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-93b23b59-4676-4148-a976-82796fecdcd3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:11:46.606346 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-178b3233-cd8f-4912-aada-54ba73cf474e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:46.607826 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-178b3233-cd8f-4912-aada-54ba73cf474e service neutron] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:46.609887 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93b23b59-4676-4148-a976-82796fecdcd3 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:46.613749 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23f3505a-a4f1-4996-96c0-a767d76659c2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:46.614362 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23f3505a-a4f1-4996-96c0-a767d76659c2 service neutron] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:46.614636 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23f3505a-a4f1-4996-96c0-a767d76659c2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:46.614935 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23f3505a-a4f1-4996-96c0-a767d76659c2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:46.617404 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 799/1612] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:11:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 134 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:11:46.618605 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e20a5e97-ce1f-4e71-899d-7709bc99c473 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:46.618916 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e20a5e97-ce1f-4e71-899d-7709bc99c473 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:46.619205 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e20a5e97-ce1f-4e71-899d-7709bc99c473 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:46.624295 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23f3505a-a4f1-4996-96c0-a767d76659c2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:46.625723 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23f3505a-a4f1-4996-96c0-a767d76659c2 service neutron] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:46.636730 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 814/1613] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:11:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:11:46.637289 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23f3505a-a4f1-4996-96c0-a767d76659c2 service neutron] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:46.639282 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-e20a5e97-ce1f-4e71-899d-7709bc99c473 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:11:46.640504 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-e20a5e97-ce1f-4e71-899d-7709bc99c473 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:11:46.646977 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23f3505a-a4f1-4996-96c0-a767d76659c2 service neutron] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:46.649600 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 800/1614] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:11:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:11:46.650234 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e20a5e97-ce1f-4e71-899d-7709bc99c473 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:46.651206 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fec95e7b-9702-4988-8f11-d30561c5bca1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:46.651843 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec95e7b-9702-4988-8f11-d30561c5bca1 service neutron] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:46.652253 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec95e7b-9702-4988-8f11-d30561c5bca1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:46.652500 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec95e7b-9702-4988-8f11-d30561c5bca1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:46.660134 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e20a5e97-ce1f-4e71-899d-7709bc99c473 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:46.663311 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fec95e7b-9702-4988-8f11-d30561c5bca1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:46.664077 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-285d91ea-e353-40d4-89ad-d3b67720795f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:46.664785 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-285d91ea-e353-40d4-89ad-d3b67720795f service neutron] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:46.664906 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fec95e7b-9702-4988-8f11-d30561c5bca1 service neutron] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:46.665087 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-285d91ea-e353-40d4-89ad-d3b67720795f service neutron] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:46.665334 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-285d91ea-e353-40d4-89ad-d3b67720795f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:46.677421 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 815/1615] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:11:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:11:46.678359 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-375f5ee1-d8ae-443c-b85f-d19e52055fb6 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:46.678697 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-375f5ee1-d8ae-443c-b85f-d19e52055fb6 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:46.678999 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-375f5ee1-d8ae-443c-b85f-d19e52055fb6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:46.680498 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-285d91ea-e353-40d4-89ad-d3b67720795f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:46.682409 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-285d91ea-e353-40d4-89ad-d3b67720795f service neutron] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:46.697665 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 801/1616] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:11:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 47 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:11:46.698955 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1b5734-96ea-42f2-86b6-4ac6adc37138 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:46.699383 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1b5734-96ea-42f2-86b6-4ac6adc37138 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:46.699756 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1b5734-96ea-42f2-86b6-4ac6adc37138 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:46.717367 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-375f5ee1-d8ae-443c-b85f-d19e52055fb6 None None] MFA Rules not processed for user `34204351339944f1995ce340ca36409b`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:11:46.719248 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-375f5ee1-d8ae-443c-b85f-d19e52055fb6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:11:46.726563 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:11:46.741065 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-ca1b5734-96ea-42f2-86b6-4ac6adc37138 None None] MFA Rules not processed for user `5f1d61f4a3a84cbbaf9fe33f3225c58e`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:11:46.742402 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-ca1b5734-96ea-42f2-86b6-4ac6adc37138 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:11:46.776150 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:11:46.821873 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 802/1617] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:11:46 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:11:46.821873 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca1b5734-96ea-42f2-86b6-4ac6adc37138 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:46.822188 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 816/1618] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:11:46 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:11:46.822188 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-375f5ee1-d8ae-443c-b85f-d19e52055fb6 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:46.834703 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-375f5ee1-d8ae-443c-b85f-d19e52055fb6 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:46.835927 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca1b5734-96ea-42f2-86b6-4ac6adc37138 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:46.839859 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb8700d8-3689-449b-82b4-5e3d282e6510 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:46.841087 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb8700d8-3689-449b-82b4-5e3d282e6510 service neutron] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:46.841376 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb8700d8-3689-449b-82b4-5e3d282e6510 service neutron] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:46.841667 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb8700d8-3689-449b-82b4-5e3d282e6510 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:46.842360 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0345d2cb-5074-4c86-a83f-de628dd02d14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:46.843276 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0345d2cb-5074-4c86-a83f-de628dd02d14 admin admin] REQUEST_METHOD: `PUT` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:46.843662 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0345d2cb-5074-4c86-a83f-de628dd02d14 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:46.844322 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0345d2cb-5074-4c86-a83f-de628dd02d14 admin admin] PATH_INFO: `/v3/projects/2c5193813eb045d5910534d3c6e6234b/users/863102c4f8c04ffa825ce2e0cae463d5/roles/9d9d5d2c270146488249b7ed943280db` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:46.853999 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb8700d8-3689-449b-82b4-5e3d282e6510 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:46.855707 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb8700d8-3689-449b-82b4-5e3d282e6510 service neutron] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:46.863410 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0345d2cb-5074-4c86-a83f-de628dd02d14 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2c5193813eb045d5910534d3c6e6234b, user_id=863102c4f8c04ffa825ce2e0cae463d5, role_id=9d9d5d2c270146488249b7ed943280db)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:46.866861 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0345d2cb-5074-4c86-a83f-de628dd02d14 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:46.867840 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 817/1619] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:11:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 47 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:11:46.869041 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb8700d8-3689-449b-82b4-5e3d282e6510 service neutron] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:46.880808 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb8700d8-3689-449b-82b4-5e3d282e6510 service neutron] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:46.885475 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 803/1620] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:11:46 2022] PUT /identity/v3/projects/2c5193813eb045d5910534d3c6e6234b/users/863102c4f8c04ffa825ce2e0cae463d5/roles/9d9d5d2c270146488249b7ed943280db => generated 0 bytes in 66 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:11:46.886623 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0345d2cb-5074-4c86-a83f-de628dd02d14 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:46.951944 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3033575-4327-4248-af8e-b0c294bd64d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:46.952791 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3033575-4327-4248-af8e-b0c294bd64d4 admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:46.953078 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3033575-4327-4248-af8e-b0c294bd64d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:46.953422 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3033575-4327-4248-af8e-b0c294bd64d4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:46.954184 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3033575-4327-4248-af8e-b0c294bd64d4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:46.957137 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3033575-4327-4248-af8e-b0c294bd64d4 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:46.970660 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0345d2cb-5074-4c86-a83f-de628dd02d14 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:46.976379 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5e596e4-c65b-4d6e-8825-552a9caba7e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:46.977043 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5e596e4-c65b-4d6e-8825-552a9caba7e3 admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:46.977324 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5e596e4-c65b-4d6e-8825-552a9caba7e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:46.977676 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5e596e4-c65b-4d6e-8825-552a9caba7e3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:46.978544 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5e596e4-c65b-4d6e-8825-552a9caba7e3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:46.981282 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5e596e4-c65b-4d6e-8825-552a9caba7e3 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:46.993813 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 818/1621] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:11:46 2022] POST /identity/v3/projects => generated 369 bytes in 125 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:11:46.994729 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3033575-4327-4248-af8e-b0c294bd64d4 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:47.017133 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 804/1622] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:11:46 2022] POST /identity/v3/projects => generated 363 bytes in 131 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:11:47.018126 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5e596e4-c65b-4d6e-8825-552a9caba7e3 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:47.117802 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3033575-4327-4248-af8e-b0c294bd64d4 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:47.147737 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5e596e4-c65b-4d6e-8825-552a9caba7e3 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:47.160304 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91190611-5574-43bc-83a0-1ee3ca1c579d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:47.160981 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91190611-5574-43bc-83a0-1ee3ca1c579d admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:47.161377 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91190611-5574-43bc-83a0-1ee3ca1c579d admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:47.161652 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91190611-5574-43bc-83a0-1ee3ca1c579d admin admin] PATH_INFO: `/v3/users` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:47.162288 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91190611-5574-43bc-83a0-1ee3ca1c579d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:47.164499 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:11:47.172061 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91190611-5574-43bc-83a0-1ee3ca1c579d admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:47.227385 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 805/1623] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:11:47 2022] POST /identity/v3/users => generated 291 bytes in 210 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:11:47.228235 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91190611-5574-43bc-83a0-1ee3ca1c579d admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:47.238542 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f73c51e6-dd9c-4853-8253-cbe033fc6063 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:47.239710 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f73c51e6-dd9c-4853-8253-cbe033fc6063 admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:47.239986 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f73c51e6-dd9c-4853-8253-cbe033fc6063 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:47.240254 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f73c51e6-dd9c-4853-8253-cbe033fc6063 admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:47.240900 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f73c51e6-dd9c-4853-8253-cbe033fc6063 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:47.242517 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f73c51e6-dd9c-4853-8253-cbe033fc6063 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:47.250171 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91190611-5574-43bc-83a0-1ee3ca1c579d admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:47.258910 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 819/1624] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:11:46 2022] GET /identity/v3/roles => generated 1420 bytes in 262 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:11:47.258910 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f73c51e6-dd9c-4853-8253-cbe033fc6063 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:47.259182 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2651df6d-0b78-4846-be39-03e5018d17f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:47.259182 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2651df6d-0b78-4846-be39-03e5018d17f8 admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:47.259649 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2651df6d-0b78-4846-be39-03e5018d17f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:47.260254 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2651df6d-0b78-4846-be39-03e5018d17f8 admin admin] PATH_INFO: `/v3/users` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:47.260957 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2651df6d-0b78-4846-be39-03e5018d17f8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:47.263286 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2651df6d-0b78-4846-be39-03e5018d17f8 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:47.270155 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f73c51e6-dd9c-4853-8253-cbe033fc6063 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:47.275346 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba1e5c9a-aff2-47d5-9816-2d78d98f4b69 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:47.276030 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba1e5c9a-aff2-47d5-9816-2d78d98f4b69 admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:47.276381 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba1e5c9a-aff2-47d5-9816-2d78d98f4b69 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:47.276725 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba1e5c9a-aff2-47d5-9816-2d78d98f4b69 admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:47.277591 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba1e5c9a-aff2-47d5-9816-2d78d98f4b69 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:47.279661 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba1e5c9a-aff2-47d5-9816-2d78d98f4b69 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:47.293860 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 820/1625] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:11:47 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:11:47.295068 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba1e5c9a-aff2-47d5-9816-2d78d98f4b69 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:47.300334 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 806/1626] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:11:47 2022] POST /identity/v3/users => generated 288 bytes in 72 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:11:47.303100 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2651df6d-0b78-4846-be39-03e5018d17f8 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:47.311809 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba1e5c9a-aff2-47d5-9816-2d78d98f4b69 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:47.313109 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:11:47.321577 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcd16525-7fa6-424b-8784-4eee0840fce4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:47.322228 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2651df6d-0b78-4846-be39-03e5018d17f8 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:47.323207 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcd16525-7fa6-424b-8784-4eee0840fce4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:47.323703 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcd16525-7fa6-424b-8784-4eee0840fce4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:47.324148 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcd16525-7fa6-424b-8784-4eee0840fce4 admin admin] PATH_INFO: `/v3/projects/2c5193813eb045d5910534d3c6e6234b/users/863102c4f8c04ffa825ce2e0cae463d5/roles/9d9d5d2c270146488249b7ed943280db` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:47.326877 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcd16525-7fa6-424b-8784-4eee0840fce4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2c5193813eb045d5910534d3c6e6234b, user_id=863102c4f8c04ffa825ce2e0cae463d5, role_id=9d9d5d2c270146488249b7ed943280db)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:47.329487 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b060f42-dcde-4a69-bfed-3bb811bf82f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:47.330408 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b060f42-dcde-4a69-bfed-3bb811bf82f8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:47.330781 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b060f42-dcde-4a69-bfed-3bb811bf82f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:47.331215 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b060f42-dcde-4a69-bfed-3bb811bf82f8 admin admin] PATH_INFO: `/v3/projects/eaf61f7b1bda4940b6b72cac26ee5cc4/users/79150f7e98254944bc2a25e41fd8e20e/roles/9d9d5d2c270146488249b7ed943280db` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:47.331487 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcd16525-7fa6-424b-8784-4eee0840fce4 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:47.357519 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 821/1627] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:11:47 2022] PUT /identity/v3/projects/2c5193813eb045d5910534d3c6e6234b/users/863102c4f8c04ffa825ce2e0cae463d5/roles/9d9d5d2c270146488249b7ed943280db => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:11:47.357519 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcd16525-7fa6-424b-8784-4eee0840fce4 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:47.376649 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b060f42-dcde-4a69-bfed-3bb811bf82f8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=eaf61f7b1bda4940b6b72cac26ee5cc4, user_id=79150f7e98254944bc2a25e41fd8e20e, role_id=9d9d5d2c270146488249b7ed943280db)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:47.381314 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b060f42-dcde-4a69-bfed-3bb811bf82f8 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:47.419033 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 807/1628] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:11:47 2022] PUT /identity/v3/projects/eaf61f7b1bda4940b6b72cac26ee5cc4/users/79150f7e98254944bc2a25e41fd8e20e/roles/9d9d5d2c270146488249b7ed943280db => generated 0 bytes in 118 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:11:47.420994 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c44ed54-f003-498d-9e85-93a0a647a571 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:47.424150 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c44ed54-f003-498d-9e85-93a0a647a571 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:47.424736 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c44ed54-f003-498d-9e85-93a0a647a571 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:47.491917 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-1c44ed54-f003-498d-9e85-93a0a647a571 None None] MFA Rules not processed for user `863102c4f8c04ffa825ce2e0cae463d5`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:11:47.493672 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-1c44ed54-f003-498d-9e85-93a0a647a571 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:11:47.498595 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcd16525-7fa6-424b-8784-4eee0840fce4 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:47.652124 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6efaa2b-3e75-4ba7-a87b-5dcab66ae5b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:47.652298 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6efaa2b-3e75-4ba7-a87b-5dcab66ae5b6 admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:47.652481 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6efaa2b-3e75-4ba7-a87b-5dcab66ae5b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:47.652745 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6efaa2b-3e75-4ba7-a87b-5dcab66ae5b6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:47.654094 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6efaa2b-3e75-4ba7-a87b-5dcab66ae5b6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:47.657176 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6efaa2b-3e75-4ba7-a87b-5dcab66ae5b6 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:47.660098 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 808/1629] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:11:47 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 240 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:11:47.661592 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c44ed54-f003-498d-9e85-93a0a647a571 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:47.679269 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c44ed54-f003-498d-9e85-93a0a647a571 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:47.679646 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 822/1630] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:11:47 2022] GET /identity/v3/roles => generated 1420 bytes in 330 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:11:47.680569 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6efaa2b-3e75-4ba7-a87b-5dcab66ae5b6 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:47.693342 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4420b07-2722-41e6-91d5-86e4a1e93ae0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:47.695071 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4420b07-2722-41e6-91d5-86e4a1e93ae0 admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:47.695638 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4420b07-2722-41e6-91d5-86e4a1e93ae0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:47.695915 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4420b07-2722-41e6-91d5-86e4a1e93ae0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:47.697745 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4420b07-2722-41e6-91d5-86e4a1e93ae0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:47.701541 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4420b07-2722-41e6-91d5-86e4a1e93ae0 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:47.730424 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 809/1631] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:11:47 2022] GET /identity/v3/roles => generated 1420 bytes in 69 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:11:47.730703 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4420b07-2722-41e6-91d5-86e4a1e93ae0 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:47.744354 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4420b07-2722-41e6-91d5-86e4a1e93ae0 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:47.752130 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71700074-dcb5-45f9-999d-2612cd7d9703 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:47.753642 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71700074-dcb5-45f9-999d-2612cd7d9703 admin admin] REQUEST_METHOD: `PUT` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:47.753642 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71700074-dcb5-45f9-999d-2612cd7d9703 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:47.754869 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71700074-dcb5-45f9-999d-2612cd7d9703 admin admin] PATH_INFO: `/v3/projects/35e2d52f0b9c45dfa6235c9ed9bbc4b6/users/f1c7e121df9f4e588790c586ceb6e7a8/roles/9d9d5d2c270146488249b7ed943280db` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:47.778995 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71700074-dcb5-45f9-999d-2612cd7d9703 admin admin] RBAC: Authorizing `identity:create_grant(project_id=35e2d52f0b9c45dfa6235c9ed9bbc4b6, user_id=f1c7e121df9f4e588790c586ceb6e7a8, role_id=9d9d5d2c270146488249b7ed943280db)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:47.783007 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71700074-dcb5-45f9-999d-2612cd7d9703 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:47.799208 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:11:47.823854 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 810/1632] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:11:47 2022] PUT /identity/v3/projects/35e2d52f0b9c45dfa6235c9ed9bbc4b6/users/f1c7e121df9f4e588790c586ceb6e7a8/roles/9d9d5d2c270146488249b7ed943280db => generated 0 bytes in 90 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:11:47.827018 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71700074-dcb5-45f9-999d-2612cd7d9703 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:47.827878 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6efaa2b-3e75-4ba7-a87b-5dcab66ae5b6 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:47.948961 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c1fb43-3b8d-4d06-b7cb-41e874a09437 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '019a79b2affe478b8a30d1c538d0415c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:47.949968 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c1fb43-3b8d-4d06-b7cb-41e874a09437 service cinder] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:47.950454 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c1fb43-3b8d-4d06-b7cb-41e874a09437 service cinder] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:47.950792 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c1fb43-3b8d-4d06-b7cb-41e874a09437 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:47.957135 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71700074-dcb5-45f9-999d-2612cd7d9703 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:47.965638 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e19e5c1-6c67-4991-b64e-8becd815bc56 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:47.967798 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e19e5c1-6c67-4991-b64e-8becd815bc56 admin admin] REQUEST_METHOD: `PUT` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:47.968287 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e19e5c1-6c67-4991-b64e-8becd815bc56 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:47.969525 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e19e5c1-6c67-4991-b64e-8becd815bc56 admin admin] PATH_INFO: `/v3/projects/eaf61f7b1bda4940b6b72cac26ee5cc4/users/79150f7e98254944bc2a25e41fd8e20e/roles/9d9d5d2c270146488249b7ed943280db` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:47.972594 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e19e5c1-6c67-4991-b64e-8becd815bc56 admin admin] RBAC: Authorizing `identity:create_grant(project_id=eaf61f7b1bda4940b6b72cac26ee5cc4, user_id=79150f7e98254944bc2a25e41fd8e20e, role_id=9d9d5d2c270146488249b7ed943280db)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:47.977370 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e19e5c1-6c67-4991-b64e-8becd815bc56 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:48.004290 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 811/1633] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:11:47 2022] PUT /identity/v3/projects/eaf61f7b1bda4940b6b72cac26ee5cc4/users/79150f7e98254944bc2a25e41fd8e20e/roles/9d9d5d2c270146488249b7ed943280db => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:11:48.006282 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e19e5c1-6c67-4991-b64e-8becd815bc56 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:48.061250 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:11:48.076674 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46c1fb43-3b8d-4d06-b7cb-41e874a09437 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:48.078144 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46c1fb43-3b8d-4d06-b7cb-41e874a09437 service cinder] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:48.096390 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 823/1634] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:11:47 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 416 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:11:48.099121 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d69ee01b-4990-47df-a256-577ea98b32c2 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:48.099121 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d69ee01b-4990-47df-a256-577ea98b32c2 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:48.099121 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d69ee01b-4990-47df-a256-577ea98b32c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:48.158029 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e19e5c1-6c67-4991-b64e-8becd815bc56 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:48.184697 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3c1c542-097e-45af-9208-818ee8b13081 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:48.185460 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3c1c542-097e-45af-9208-818ee8b13081 admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:48.185615 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3c1c542-097e-45af-9208-818ee8b13081 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:48.185888 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3c1c542-097e-45af-9208-818ee8b13081 admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:48.186606 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3c1c542-097e-45af-9208-818ee8b13081 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:48.193450 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3c1c542-097e-45af-9208-818ee8b13081 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:48.211745 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-d69ee01b-4990-47df-a256-577ea98b32c2 None None] MFA Rules not processed for user `79150f7e98254944bc2a25e41fd8e20e`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:11:48.213041 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-d69ee01b-4990-47df-a256-577ea98b32c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:11:48.224266 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 812/1635] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:11:48 2022] GET /identity/v3/roles => generated 1420 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:11:48.225517 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8046cd1-86e8-4848-9d0b-6ff17289a08b None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:48.225806 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8046cd1-86e8-4848-9d0b-6ff17289a08b None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:48.236526 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8046cd1-86e8-4848-9d0b-6ff17289a08b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:48.334129 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-f8046cd1-86e8-4848-9d0b-6ff17289a08b None None] MFA Rules not processed for user `e6c0a8421dcd402cbc46505e4b292cce`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:11:48.335542 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-f8046cd1-86e8-4848-9d0b-6ff17289a08b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:11:48.403560 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 824/1636] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:11:48 2022] POST /identity/v3/auth/tokens => generated 3313 bytes in 305 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:11:48.404110 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d69ee01b-4990-47df-a256-577ea98b32c2 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:48.426419 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d69ee01b-4990-47df-a256-577ea98b32c2 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:48.437107 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e3c208b-7c27-421f-9ce5-4e7d3d84f5b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:48.437766 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e3c208b-7c27-421f-9ce5-4e7d3d84f5b0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:48.438045 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e3c208b-7c27-421f-9ce5-4e7d3d84f5b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:48.438346 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e3c208b-7c27-421f-9ce5-4e7d3d84f5b0 admin admin] PATH_INFO: `/v3/projects/35e2d52f0b9c45dfa6235c9ed9bbc4b6/users/f1c7e121df9f4e588790c586ceb6e7a8/roles/9d9d5d2c270146488249b7ed943280db` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:48.441273 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e3c208b-7c27-421f-9ce5-4e7d3d84f5b0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=35e2d52f0b9c45dfa6235c9ed9bbc4b6, user_id=f1c7e121df9f4e588790c586ceb6e7a8, role_id=9d9d5d2c270146488249b7ed943280db)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:48.444759 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e3c208b-7c27-421f-9ce5-4e7d3d84f5b0 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:48.468867 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 825/1637] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:11:48 2022] PUT /identity/v3/projects/35e2d52f0b9c45dfa6235c9ed9bbc4b6/users/f1c7e121df9f4e588790c586ceb6e7a8/roles/9d9d5d2c270146488249b7ed943280db => generated 0 bytes in 65 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:11:48.469920 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e3c208b-7c27-421f-9ce5-4e7d3d84f5b0 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:48.477508 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 813/1638] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:11:48 2022] POST /identity/v3/auth/tokens => generated 3282 bytes in 253 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:11:48.478397 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8046cd1-86e8-4848-9d0b-6ff17289a08b None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:48.515358 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:11:48.574116 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e3c208b-7c27-421f-9ce5-4e7d3d84f5b0 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:48.577965 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a792551-4276-49ce-8679-dc07d95f791f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '019a79b2affe478b8a30d1c538d0415c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:48.578947 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a792551-4276-49ce-8679-dc07d95f791f service cinder] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:48.579229 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a792551-4276-49ce-8679-dc07d95f791f service cinder] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:48.579502 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a792551-4276-49ce-8679-dc07d95f791f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:48.592429 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8046cd1-86e8-4848-9d0b-6ff17289a08b None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:48.601559 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff3687ec-7d81-474f-b8f6-be7e58f3ec31 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '019a79b2affe478b8a30d1c538d0415c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:48.602201 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3687ec-7d81-474f-b8f6-be7e58f3ec31 service cinder] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:48.602480 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3687ec-7d81-474f-b8f6-be7e58f3ec31 service cinder] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:48.602751 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3687ec-7d81-474f-b8f6-be7e58f3ec31 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:48.678227 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a792551-4276-49ce-8679-dc07d95f791f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:48.680278 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a792551-4276-49ce-8679-dc07d95f791f service cinder] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:48.694830 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 826/1639] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:11:48 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 225 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:11:48.696266 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d11ee8b-30fd-4d97-8c98-12b79c51e1bf None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:48.696902 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d11ee8b-30fd-4d97-8c98-12b79c51e1bf None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:48.696902 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d11ee8b-30fd-4d97-8c98-12b79c51e1bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:48.735240 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff3687ec-7d81-474f-b8f6-be7e58f3ec31 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:48.736765 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff3687ec-7d81-474f-b8f6-be7e58f3ec31 service cinder] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:48.743484 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:11:48.747954 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-8d11ee8b-30fd-4d97-8c98-12b79c51e1bf None None] MFA Rules not processed for user `f1c7e121df9f4e588790c586ceb6e7a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:11:48.752315 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-8d11ee8b-30fd-4d97-8c98-12b79c51e1bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:11:48.763288 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 814/1640] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:11:48 2022] GET /identity/v3/auth/tokens => generated 3313 bytes in 285 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:11:48.764662 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff3687ec-7d81-474f-b8f6-be7e58f3ec31 service cinder] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:48.971437 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff3687ec-7d81-474f-b8f6-be7e58f3ec31 service cinder] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:48.992337 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ac98426-5bcb-416a-8171-24b707fb13a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:48.997703 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ac98426-5bcb-416a-8171-24b707fb13a7 admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:48.998494 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ac98426-5bcb-416a-8171-24b707fb13a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:48.998898 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ac98426-5bcb-416a-8171-24b707fb13a7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:49.000214 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ac98426-5bcb-416a-8171-24b707fb13a7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:49.004026 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 827/1641] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:11:48 2022] POST /identity/v3/auth/tokens => generated 3307 bytes in 308 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:11:49.011739 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ac98426-5bcb-416a-8171-24b707fb13a7 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:49.063189 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d11ee8b-30fd-4d97-8c98-12b79c51e1bf None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:49.082620 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d11ee8b-30fd-4d97-8c98-12b79c51e1bf None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:49.091647 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a2b4eef-0636-43f3-99d9-d290f5db7c72 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '019a79b2affe478b8a30d1c538d0415c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:49.095870 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a2b4eef-0636-43f3-99d9-d290f5db7c72 service cinder] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:49.097085 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a2b4eef-0636-43f3-99d9-d290f5db7c72 service cinder] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:49.097535 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a2b4eef-0636-43f3-99d9-d290f5db7c72 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:49.107394 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 815/1642] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:11:48 2022] POST /identity/v3/projects => generated 405 bytes in 343 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:11:49.112013 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ac98426-5bcb-416a-8171-24b707fb13a7 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:49.113099 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a2b4eef-0636-43f3-99d9-d290f5db7c72 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:49.116123 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a2b4eef-0636-43f3-99d9-d290f5db7c72 service cinder] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:49.147286 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 828/1643] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:11:49 2022] GET /identity/v3/auth/tokens => generated 3307 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:11:49.326474 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ac98426-5bcb-416a-8171-24b707fb13a7 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:49.334404 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d877e74-49fc-4f12-a948-40ec4aa018c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:49.335268 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d877e74-49fc-4f12-a948-40ec4aa018c3 admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:49.335651 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d877e74-49fc-4f12-a948-40ec4aa018c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:49.336026 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d877e74-49fc-4f12-a948-40ec4aa018c3 admin admin] PATH_INFO: `/v3/users` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:49.337249 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d877e74-49fc-4f12-a948-40ec4aa018c3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:49.339675 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d877e74-49fc-4f12-a948-40ec4aa018c3 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:49.370383 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 816/1644] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:11:49 2022] POST /identity/v3/users => generated 309 bytes in 258 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:11:49.371895 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a2b4eef-0636-43f3-99d9-d290f5db7c72 service cinder] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:49.387324 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a2b4eef-0636-43f3-99d9-d290f5db7c72 service cinder] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:49.401972 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa4856d4-676b-40b8-80e0-2c33d7afa656 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:49.401972 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa4856d4-676b-40b8-80e0-2c33d7afa656 admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:49.401972 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa4856d4-676b-40b8-80e0-2c33d7afa656 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:49.401972 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa4856d4-676b-40b8-80e0-2c33d7afa656 admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:49.402446 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa4856d4-676b-40b8-80e0-2c33d7afa656 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:49.402446 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa4856d4-676b-40b8-80e0-2c33d7afa656 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:49.422268 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 829/1645] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:11:49 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:11:49.423374 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d877e74-49fc-4f12-a948-40ec4aa018c3 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:49.443727 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa4856d4-676b-40b8-80e0-2c33d7afa656 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:49.465123 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa4856d4-676b-40b8-80e0-2c33d7afa656 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:49.482791 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf54ab15-6271-4f31-b8be-6baddce7c7df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:49.483490 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf54ab15-6271-4f31-b8be-6baddce7c7df admin admin] REQUEST_METHOD: `PUT` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:49.483773 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf54ab15-6271-4f31-b8be-6baddce7c7df admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:49.484050 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf54ab15-6271-4f31-b8be-6baddce7c7df admin admin] PATH_INFO: `/v3/projects/572ca9b0240149b3860a3f827ecc74eb/users/c9febd36c47e410a826629a43bd4871f/roles/9d9d5d2c270146488249b7ed943280db` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:49.519752 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf54ab15-6271-4f31-b8be-6baddce7c7df admin admin] RBAC: Authorizing `identity:create_grant(project_id=572ca9b0240149b3860a3f827ecc74eb, user_id=c9febd36c47e410a826629a43bd4871f, role_id=9d9d5d2c270146488249b7ed943280db)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:49.522776 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf54ab15-6271-4f31-b8be-6baddce7c7df admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:49.545344 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d877e74-49fc-4f12-a948-40ec4aa018c3 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:49.552392 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 830/1646] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:11:49 2022] PUT /identity/v3/projects/572ca9b0240149b3860a3f827ecc74eb/users/c9febd36c47e410a826629a43bd4871f/roles/9d9d5d2c270146488249b7ed943280db => generated 0 bytes in 127 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:11:49.556978 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf54ab15-6271-4f31-b8be-6baddce7c7df admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:49.576583 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:11:49.588784 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:11:49.657739 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b44e0c46-4336-4137-b2ef-e56f74f1f339 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '019a79b2affe478b8a30d1c538d0415c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:49.659355 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b44e0c46-4336-4137-b2ef-e56f74f1f339 service cinder] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:49.662588 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b44e0c46-4336-4137-b2ef-e56f74f1f339 service cinder] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:49.663194 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b44e0c46-4336-4137-b2ef-e56f74f1f339 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:49.711716 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf54ab15-6271-4f31-b8be-6baddce7c7df admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:49.729661 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b98e4c12-ae42-448d-b405-45ea62c5492f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:49.731124 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b98e4c12-ae42-448d-b405-45ea62c5492f admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:49.731837 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b98e4c12-ae42-448d-b405-45ea62c5492f admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:49.731837 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b98e4c12-ae42-448d-b405-45ea62c5492f admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:49.732368 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b98e4c12-ae42-448d-b405-45ea62c5492f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:49.733990 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b98e4c12-ae42-448d-b405-45ea62c5492f admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:49.766865 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 831/1647] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:11:49 2022] GET /identity/v3/roles => generated 1420 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 05:11:49.769981 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b98e4c12-ae42-448d-b405-45ea62c5492f admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:49.793430 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b44e0c46-4336-4137-b2ef-e56f74f1f339 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:49.796442 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b44e0c46-4336-4137-b2ef-e56f74f1f339 service cinder] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:49.806001 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b98e4c12-ae42-448d-b405-45ea62c5492f admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:49.825216 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05b4e804-8ebd-4b88-a273-1d9e698a3e7c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:49.827549 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b4e804-8ebd-4b88-a273-1d9e698a3e7c admin admin] REQUEST_METHOD: `PUT` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:49.827767 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b4e804-8ebd-4b88-a273-1d9e698a3e7c admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:49.828047 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b4e804-8ebd-4b88-a273-1d9e698a3e7c admin admin] PATH_INFO: `/v3/projects/572ca9b0240149b3860a3f827ecc74eb/users/c9febd36c47e410a826629a43bd4871f/roles/9d9d5d2c270146488249b7ed943280db` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:49.830892 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 817/1648] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:11:49 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 408 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:11:49.832420 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05b4e804-8ebd-4b88-a273-1d9e698a3e7c admin admin] RBAC: Authorizing `identity:create_grant(project_id=572ca9b0240149b3860a3f827ecc74eb, user_id=c9febd36c47e410a826629a43bd4871f, role_id=9d9d5d2c270146488249b7ed943280db)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:49.835461 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05b4e804-8ebd-4b88-a273-1d9e698a3e7c admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:49.856607 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 832/1649] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:11:49 2022] PUT /identity/v3/projects/572ca9b0240149b3860a3f827ecc74eb/users/c9febd36c47e410a826629a43bd4871f/roles/9d9d5d2c270146488249b7ed943280db => generated 0 bytes in 87 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:11:51.631770 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8096cb37-14d5-49fa-9f82-5467f9b5e447 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:51.632373 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8096cb37-14d5-49fa-9f82-5467f9b5e447 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:51.632373 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8096cb37-14d5-49fa-9f82-5467f9b5e447 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:51.692455 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-8096cb37-14d5-49fa-9f82-5467f9b5e447 None None] MFA Rules not processed for user `c9febd36c47e410a826629a43bd4871f`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:11:51.693854 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-8096cb37-14d5-49fa-9f82-5467f9b5e447 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:11:51.788429 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 818/1650] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:11:51 2022] POST /identity/v3/auth/tokens => generated 3349 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 05:11:52.199033 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05b4e804-8ebd-4b88-a273-1d9e698a3e7c admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:52.349109 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05b4e804-8ebd-4b88-a273-1d9e698a3e7c admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:52.365175 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b78b2c00-4fad-4248-b7dd-648014581b7d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:52.365829 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b78b2c00-4fad-4248-b7dd-648014581b7d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:52.366102 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b78b2c00-4fad-4248-b7dd-648014581b7d admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:52.366374 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b78b2c00-4fad-4248-b7dd-648014581b7d admin admin] PATH_INFO: `/v3/users/5f1d61f4a3a84cbbaf9fe33f3225c58e` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:52.367942 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b78b2c00-4fad-4248-b7dd-648014581b7d admin admin] RBAC: Authorizing `identity:delete_user(user_id=5f1d61f4a3a84cbbaf9fe33f3225c58e)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:52.369478 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b78b2c00-4fad-4248-b7dd-648014581b7d admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:52.508811 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-b78b2c00-4fad-4248-b7dd-648014581b7d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5f1d61f4a3a84cbbaf9fe33f3225c58e'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:11:52.546582 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-b78b2c00-4fad-4248-b7dd-648014581b7d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5f1d61f4a3a84cbbaf9fe33f3225c58e'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:11:52.575560 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-b78b2c00-4fad-4248-b7dd-648014581b7d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5f1d61f4a3a84cbbaf9fe33f3225c58e'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:11:52.613245 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-b78b2c00-4fad-4248-b7dd-648014581b7d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5f1d61f4a3a84cbbaf9fe33f3225c58e'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:11:52.614084 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-b78b2c00-4fad-4248-b7dd-648014581b7d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5f1d61f4a3a84cbbaf9fe33f3225c58e'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:11:52.636084 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 833/1651] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:11:52 2022] DELETE /identity/v3/users/5f1d61f4a3a84cbbaf9fe33f3225c58e => generated 0 bytes in 437 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:11:52.641249 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dce3c2f0-4b43-4077-bb36-487b895fb1b4 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:52.642139 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dce3c2f0-4b43-4077-bb36-487b895fb1b4 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:52.642665 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dce3c2f0-4b43-4077-bb36-487b895fb1b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:52.709026 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-dce3c2f0-4b43-4077-bb36-487b895fb1b4 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:11:52.712273 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-dce3c2f0-4b43-4077-bb36-487b895fb1b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:11:52.814453 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 819/1652] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:11:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:11:52.870665 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b78b2c00-4fad-4248-b7dd-648014581b7d admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:52.873457 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-b78b2c00-4fad-4248-b7dd-648014581b7d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:11:52.899183 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:11:53.028880 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b78b2c00-4fad-4248-b7dd-648014581b7d admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:53.036960 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eaff5cb-d190-4856-9f35-8c984b006dea service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:53.037592 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eaff5cb-d190-4856-9f35-8c984b006dea service neutron] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:53.037869 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eaff5cb-d190-4856-9f35-8c984b006dea service neutron] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:53.038136 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eaff5cb-d190-4856-9f35-8c984b006dea service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:53.065716 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eaff5cb-d190-4856-9f35-8c984b006dea service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:53.068195 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eaff5cb-d190-4856-9f35-8c984b006dea service neutron] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:53.090579 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 834/1653] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:11:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 220 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:11:54.286255 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c68b8e9-ee94-45b7-a715-be425d19d051 None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:54.287307 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c68b8e9-ee94-45b7-a715-be425d19d051 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:54.287586 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c68b8e9-ee94-45b7-a715-be425d19d051 None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:54.289975 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 820/1654] 10.222.0.77 () {54 vars in 921 bytes} [Tue Aug 30 05:11:54 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:11:54.374398 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b49ddb82-7c36-485a-9843-5e089993e17e None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:54.374877 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b49ddb82-7c36-485a-9843-5e089993e17e None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:54.375138 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b49ddb82-7c36-485a-9843-5e089993e17e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:54.437895 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-b49ddb82-7c36-485a-9843-5e089993e17e None None] MFA Rules not processed for user `1972959402704b89925001cd8d90cb0e`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:11:54.439178 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-b49ddb82-7c36-485a-9843-5e089993e17e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:11:54.706854 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 835/1655] 10.222.0.77 () {58 vars in 1018 bytes} [Tue Aug 30 05:11:54 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 333 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:11:54.877684 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c68b8e9-ee94-45b7-a715-be425d19d051 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:54.879699 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-2c68b8e9-ee94-45b7-a715-be425d19d051 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:11:54.913795 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c68b8e9-ee94-45b7-a715-be425d19d051 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:54.933044 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-432b0eab-13d3-42e5-b59d-e0a7482536ec service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1972959402704b89925001cd8d90cb0e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:54.937935 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-432b0eab-13d3-42e5-b59d-e0a7482536ec service placement] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:54.938446 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-432b0eab-13d3-42e5-b59d-e0a7482536ec service placement] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:54.939263 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-432b0eab-13d3-42e5-b59d-e0a7482536ec service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:54.958129 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:11:54.961960 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-432b0eab-13d3-42e5-b59d-e0a7482536ec service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:54.963901 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-432b0eab-13d3-42e5-b59d-e0a7482536ec service placement] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:54.996025 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 821/1656] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:11:54 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 119 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:11:56.209817 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b49ddb82-7c36-485a-9843-5e089993e17e None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:56.211392 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-b49ddb82-7c36-485a-9843-5e089993e17e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:11:56.241251 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b49ddb82-7c36-485a-9843-5e089993e17e None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:56.249854 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7a8322d-1fa2-46b0-ae31-3a5a084245b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:56.251779 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7a8322d-1fa2-46b0-ae31-3a5a084245b1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:56.252073 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7a8322d-1fa2-46b0-ae31-3a5a084245b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:56.255230 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7a8322d-1fa2-46b0-ae31-3a5a084245b1 admin admin] PATH_INFO: `/v3/users/e6c0a8421dcd402cbc46505e4b292cce` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:56.258673 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7a8322d-1fa2-46b0-ae31-3a5a084245b1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e6c0a8421dcd402cbc46505e4b292cce)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:56.264631 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7a8322d-1fa2-46b0-ae31-3a5a084245b1 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:56.382875 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-c7a8322d-1fa2-46b0-ae31-3a5a084245b1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e6c0a8421dcd402cbc46505e4b292cce'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:11:56.409429 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-c7a8322d-1fa2-46b0-ae31-3a5a084245b1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e6c0a8421dcd402cbc46505e4b292cce'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:11:56.455039 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-c7a8322d-1fa2-46b0-ae31-3a5a084245b1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e6c0a8421dcd402cbc46505e4b292cce'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:11:56.481897 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-c7a8322d-1fa2-46b0-ae31-3a5a084245b1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e6c0a8421dcd402cbc46505e4b292cce'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:11:56.482840 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-c7a8322d-1fa2-46b0-ae31-3a5a084245b1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e6c0a8421dcd402cbc46505e4b292cce'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:11:56.495315 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 836/1657] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:11:56 2022] DELETE /identity/v3/users/e6c0a8421dcd402cbc46505e4b292cce => generated 0 bytes in 285 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:11:56.502664 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-432b0eab-13d3-42e5-b59d-e0a7482536ec service placement] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:56.504691 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-432b0eab-13d3-42e5-b59d-e0a7482536ec service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:11:56.561080 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-432b0eab-13d3-42e5-b59d-e0a7482536ec service placement] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:56.682790 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3043c4bb-94bc-40ee-a97e-ce8063c14bd6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:56.683398 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3043c4bb-94bc-40ee-a97e-ce8063c14bd6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:56.683677 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3043c4bb-94bc-40ee-a97e-ce8063c14bd6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:56.683950 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3043c4bb-94bc-40ee-a97e-ce8063c14bd6 admin admin] PATH_INFO: `/v3/projects/27a07b7f74cc48e29883a106e15996a8` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:56.688967 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3043c4bb-94bc-40ee-a97e-ce8063c14bd6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=27a07b7f74cc48e29883a106e15996a8)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:56.691414 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3043c4bb-94bc-40ee-a97e-ce8063c14bd6 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:56.821139 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-3043c4bb-94bc-40ee-a97e-ce8063c14bd6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '27a07b7f74cc48e29883a106e15996a8', 'request_id': 'req-3043c4bb-94bc-40ee-a97e-ce8063c14bd6'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:11:56.831822 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-3043c4bb-94bc-40ee-a97e-ce8063c14bd6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '27a07b7f74cc48e29883a106e15996a8', 'request_id': 'req-3043c4bb-94bc-40ee-a97e-ce8063c14bd6'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:11:56.841455 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-3043c4bb-94bc-40ee-a97e-ce8063c14bd6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '27a07b7f74cc48e29883a106e15996a8', 'request_id': 'req-3043c4bb-94bc-40ee-a97e-ce8063c14bd6'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:11:56.857962 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-3043c4bb-94bc-40ee-a97e-ce8063c14bd6 admin admin] The token cache is being invalidate because project 27a07b7f74cc48e29883a106e15996a8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81624) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:11:56.859049 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-3043c4bb-94bc-40ee-a97e-ce8063c14bd6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:11:56.859844 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-3043c4bb-94bc-40ee-a97e-ce8063c14bd6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:11:56.862033 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 822/1658] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:11:56 2022] DELETE /identity/v3/projects/27a07b7f74cc48e29883a106e15996a8 => generated 0 bytes in 360 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:11:56.865796 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7a8322d-1fa2-46b0-ae31-3a5a084245b1 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:56.867472 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-c7a8322d-1fa2-46b0-ae31-3a5a084245b1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:11:56.892985 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7a8322d-1fa2-46b0-ae31-3a5a084245b1 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:56.939185 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:11:57.024311 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a3aab45-fea2-458c-b0d5-ce1204b7a3f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:57.024978 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a3aab45-fea2-458c-b0d5-ce1204b7a3f2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:57.025252 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a3aab45-fea2-458c-b0d5-ce1204b7a3f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:57.025523 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a3aab45-fea2-458c-b0d5-ce1204b7a3f2 admin admin] PATH_INFO: `/v3/users/79150f7e98254944bc2a25e41fd8e20e` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:57.030204 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a3aab45-fea2-458c-b0d5-ce1204b7a3f2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=79150f7e98254944bc2a25e41fd8e20e)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:57.033321 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a3aab45-fea2-458c-b0d5-ce1204b7a3f2 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:57.149295 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-1a3aab45-fea2-458c-b0d5-ce1204b7a3f2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '79150f7e98254944bc2a25e41fd8e20e'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:11:57.169162 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-1a3aab45-fea2-458c-b0d5-ce1204b7a3f2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '79150f7e98254944bc2a25e41fd8e20e'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:11:57.203801 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-1a3aab45-fea2-458c-b0d5-ce1204b7a3f2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '79150f7e98254944bc2a25e41fd8e20e'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:11:57.227341 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-1a3aab45-fea2-458c-b0d5-ce1204b7a3f2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '79150f7e98254944bc2a25e41fd8e20e'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:11:57.228288 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-1a3aab45-fea2-458c-b0d5-ce1204b7a3f2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '79150f7e98254944bc2a25e41fd8e20e'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:11:57.238407 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 837/1659] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:11:56 2022] DELETE /identity/v3/users/79150f7e98254944bc2a25e41fd8e20e => generated 0 bytes in 370 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:11:57.721844 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3043c4bb-94bc-40ee-a97e-ce8063c14bd6 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:57.723729 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-3043c4bb-94bc-40ee-a97e-ce8063c14bd6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:11:57.747080 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3043c4bb-94bc-40ee-a97e-ce8063c14bd6 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:57.860260 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-687f34e3-803f-4133-ba4a-57ca46f44988 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:57.860924 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-687f34e3-803f-4133-ba4a-57ca46f44988 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:57.861399 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-687f34e3-803f-4133-ba4a-57ca46f44988 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:57.861836 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-687f34e3-803f-4133-ba4a-57ca46f44988 admin admin] PATH_INFO: `/v3/projects/30239b4a7a034b13b01b8572a2ec40cb` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:57.863209 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-687f34e3-803f-4133-ba4a-57ca46f44988 admin admin] RBAC: Authorizing `identity:delete_project(project_id=30239b4a7a034b13b01b8572a2ec40cb)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:57.867319 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-687f34e3-803f-4133-ba4a-57ca46f44988 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:57.957801 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:11:57.999139 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-687f34e3-803f-4133-ba4a-57ca46f44988 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '30239b4a7a034b13b01b8572a2ec40cb', 'request_id': 'req-687f34e3-803f-4133-ba4a-57ca46f44988'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:11:58.023327 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-687f34e3-803f-4133-ba4a-57ca46f44988 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '30239b4a7a034b13b01b8572a2ec40cb', 'request_id': 'req-687f34e3-803f-4133-ba4a-57ca46f44988'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:11:58.038407 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-687f34e3-803f-4133-ba4a-57ca46f44988 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '30239b4a7a034b13b01b8572a2ec40cb', 'request_id': 'req-687f34e3-803f-4133-ba4a-57ca46f44988'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:11:58.064005 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-687f34e3-803f-4133-ba4a-57ca46f44988 admin admin] The token cache is being invalidate because project 30239b4a7a034b13b01b8572a2ec40cb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81624) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:11:58.064645 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-687f34e3-803f-4133-ba4a-57ca46f44988 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:11:58.066054 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-687f34e3-803f-4133-ba4a-57ca46f44988 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:11:58.069288 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 823/1660] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:11:57 2022] DELETE /identity/v3/projects/30239b4a7a034b13b01b8572a2ec40cb => generated 0 bytes in 348 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:11:58.073072 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a3aab45-fea2-458c-b0d5-ce1204b7a3f2 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:58.075564 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-1a3aab45-fea2-458c-b0d5-ce1204b7a3f2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:11:58.100538 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a3aab45-fea2-458c-b0d5-ce1204b7a3f2 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:58.275548 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49cfd395-51f1-49ca-ab70-9deba67204a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:58.276202 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49cfd395-51f1-49ca-ab70-9deba67204a4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:58.276475 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49cfd395-51f1-49ca-ab70-9deba67204a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:58.276748 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49cfd395-51f1-49ca-ab70-9deba67204a4 admin admin] PATH_INFO: `/v3/users/c9febd36c47e410a826629a43bd4871f` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:58.280259 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49cfd395-51f1-49ca-ab70-9deba67204a4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c9febd36c47e410a826629a43bd4871f)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:58.283398 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49cfd395-51f1-49ca-ab70-9deba67204a4 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:58.409797 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=82,keystone:DELETE=17,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:11:58.422978 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-49cfd395-51f1-49ca-ab70-9deba67204a4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c9febd36c47e410a826629a43bd4871f'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:11:58.440175 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-49cfd395-51f1-49ca-ab70-9deba67204a4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c9febd36c47e410a826629a43bd4871f'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:11:58.498442 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-49cfd395-51f1-49ca-ab70-9deba67204a4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c9febd36c47e410a826629a43bd4871f'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:11:58.509683 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-49cfd395-51f1-49ca-ab70-9deba67204a4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c9febd36c47e410a826629a43bd4871f'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:11:58.510691 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-49cfd395-51f1-49ca-ab70-9deba67204a4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c9febd36c47e410a826629a43bd4871f'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:11:58.522301 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 838/1661] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:11:58 2022] DELETE /identity/v3/users/c9febd36c47e410a826629a43bd4871f => generated 0 bytes in 450 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:11:59.209329 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-687f34e3-803f-4133-ba4a-57ca46f44988 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:59.211740 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-687f34e3-803f-4133-ba4a-57ca46f44988 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:11:59.249310 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-687f34e3-803f-4133-ba4a-57ca46f44988 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:59.437039 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49cfd395-51f1-49ca-ab70-9deba67204a4 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:11:59.438700 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-49cfd395-51f1-49ca-ab70-9deba67204a4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:11:59.460694 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1be462eb-2c65-408b-80de-f9bc8b6ad97a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:59.462486 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be462eb-2c65-408b-80de-f9bc8b6ad97a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:59.464131 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be462eb-2c65-408b-80de-f9bc8b6ad97a admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:59.464457 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be462eb-2c65-408b-80de-f9bc8b6ad97a admin admin] PATH_INFO: `/v3/projects/eaf61f7b1bda4940b6b72cac26ee5cc4` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:59.467785 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1be462eb-2c65-408b-80de-f9bc8b6ad97a admin admin] RBAC: Authorizing `identity:delete_project(project_id=eaf61f7b1bda4940b6b72cac26ee5cc4)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:59.487499 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1be462eb-2c65-408b-80de-f9bc8b6ad97a admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:59.638708 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49cfd395-51f1-49ca-ab70-9deba67204a4 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:11:59.667912 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-1be462eb-2c65-408b-80de-f9bc8b6ad97a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'eaf61f7b1bda4940b6b72cac26ee5cc4', 'request_id': 'req-1be462eb-2c65-408b-80de-f9bc8b6ad97a'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:11:59.698576 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-1be462eb-2c65-408b-80de-f9bc8b6ad97a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'eaf61f7b1bda4940b6b72cac26ee5cc4', 'request_id': 'req-1be462eb-2c65-408b-80de-f9bc8b6ad97a'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:11:59.710445 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac8d43e5-1cd6-4002-b83e-af7007a542bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:11:59.711209 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac8d43e5-1cd6-4002-b83e-af7007a542bd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:59.711420 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac8d43e5-1cd6-4002-b83e-af7007a542bd admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:59.711687 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac8d43e5-1cd6-4002-b83e-af7007a542bd admin admin] PATH_INFO: `/v3/projects/572ca9b0240149b3860a3f827ecc74eb` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:59.713249 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac8d43e5-1cd6-4002-b83e-af7007a542bd admin admin] RBAC: Authorizing `identity:delete_project(project_id=572ca9b0240149b3860a3f827ecc74eb)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:11:59.714002 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: WARNING py.warnings [None req-ac8d43e5-1cd6-4002-b83e-af7007a542bd admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 30 05:11:59.714002 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: warnings.warn(msg) Aug 30 05:11:59.714002 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: Aug 30 05:11:59.715261 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac8d43e5-1cd6-4002-b83e-af7007a542bd admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:11:59.722998 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-1be462eb-2c65-408b-80de-f9bc8b6ad97a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'eaf61f7b1bda4940b6b72cac26ee5cc4', 'request_id': 'req-1be462eb-2c65-408b-80de-f9bc8b6ad97a'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:11:59.791436 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-1be462eb-2c65-408b-80de-f9bc8b6ad97a admin admin] The token cache is being invalidate because project eaf61f7b1bda4940b6b72cac26ee5cc4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81624) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:11:59.794394 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-1be462eb-2c65-408b-80de-f9bc8b6ad97a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:11:59.800330 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-1be462eb-2c65-408b-80de-f9bc8b6ad97a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:11:59.807278 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 824/1662] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:11:59 2022] DELETE /identity/v3/projects/eaf61f7b1bda4940b6b72cac26ee5cc4 => generated 0 bytes in 597 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:11:59.808803 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f89b00fc-96e7-455c-bb6a-3c01789fc0bf None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:11:59.809100 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f89b00fc-96e7-455c-bb6a-3c01789fc0bf None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:11:59.809914 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f89b00fc-96e7-455c-bb6a-3c01789fc0bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:11:59.866687 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-ac8d43e5-1cd6-4002-b83e-af7007a542bd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '572ca9b0240149b3860a3f827ecc74eb', 'request_id': 'req-ac8d43e5-1cd6-4002-b83e-af7007a542bd'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:11:59.886949 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-f89b00fc-96e7-455c-bb6a-3c01789fc0bf None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:11:59.889202 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-f89b00fc-96e7-455c-bb6a-3c01789fc0bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:11:59.894891 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-ac8d43e5-1cd6-4002-b83e-af7007a542bd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '572ca9b0240149b3860a3f827ecc74eb', 'request_id': 'req-ac8d43e5-1cd6-4002-b83e-af7007a542bd'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:11:59.953531 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-ac8d43e5-1cd6-4002-b83e-af7007a542bd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '572ca9b0240149b3860a3f827ecc74eb', 'request_id': 'req-ac8d43e5-1cd6-4002-b83e-af7007a542bd'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:11:59.990048 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:11:59.998197 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-ac8d43e5-1cd6-4002-b83e-af7007a542bd admin admin] The token cache is being invalidate because project 572ca9b0240149b3860a3f827ecc74eb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81625) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:11:59.998728 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-ac8d43e5-1cd6-4002-b83e-af7007a542bd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:11:59.999591 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-ac8d43e5-1cd6-4002-b83e-af7007a542bd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:00.002521 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 839/1663] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:11:59 2022] DELETE /identity/v3/projects/572ca9b0240149b3860a3f827ecc74eb => generated 0 bytes in 567 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 05:12:00.012933 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2480a97b-cdd6-4e16-acae-6ff60d0140f0 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:00.013481 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2480a97b-cdd6-4e16-acae-6ff60d0140f0 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:00.013966 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2480a97b-cdd6-4e16-acae-6ff60d0140f0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:00.052596 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:00.083570 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-2480a97b-cdd6-4e16-acae-6ff60d0140f0 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:12:00.087124 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-2480a97b-cdd6-4e16-acae-6ff60d0140f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:00.115577 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 840/1664] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:12:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:12:00.117495 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7457ed5-906a-4dd0-a528-bfd565285ee3 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:00.118605 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7457ed5-906a-4dd0-a528-bfd565285ee3 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:00.119147 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7457ed5-906a-4dd0-a528-bfd565285ee3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:00.123744 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 825/1665] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:11:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 316 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:12:00.133270 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-281d553d-380e-4452-8137-adf25a534c9f None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:00.133563 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-281d553d-380e-4452-8137-adf25a534c9f None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:00.133823 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-281d553d-380e-4452-8137-adf25a534c9f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:00.157781 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-b7457ed5-906a-4dd0-a528-bfd565285ee3 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:12:00.160373 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-b7457ed5-906a-4dd0-a528-bfd565285ee3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:00.186728 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 841/1666] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:12:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 67 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:12:00.186954 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-281d553d-380e-4452-8137-adf25a534c9f None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:12:00.186954 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-281d553d-380e-4452-8137-adf25a534c9f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:00.189190 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7457ed5-906a-4dd0-a528-bfd565285ee3 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:00.214188 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7457ed5-906a-4dd0-a528-bfd565285ee3 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:00.217035 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 826/1667] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:12:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 05:12:00.217939 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-281d553d-380e-4452-8137-adf25a534c9f None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:00.230571 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f65f9315-d2db-4c2d-8f0b-e7408bcd7201 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:00.231222 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f65f9315-d2db-4c2d-8f0b-e7408bcd7201 admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:00.231499 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f65f9315-d2db-4c2d-8f0b-e7408bcd7201 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:00.231766 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f65f9315-d2db-4c2d-8f0b-e7408bcd7201 admin admin] PATH_INFO: `/v3/domains` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:00.232426 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f65f9315-d2db-4c2d-8f0b-e7408bcd7201 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:12:00.232720 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f65f9315-d2db-4c2d-8f0b-e7408bcd7201 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:00.234334 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f65f9315-d2db-4c2d-8f0b-e7408bcd7201 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:00.237028 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-281d553d-380e-4452-8137-adf25a534c9f None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:00.249337 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fb92f02-4d00-474e-8215-133249852326 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:00.250214 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fb92f02-4d00-474e-8215-133249852326 admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:00.250637 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fb92f02-4d00-474e-8215-133249852326 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:00.251167 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fb92f02-4d00-474e-8215-133249852326 admin admin] PATH_INFO: `/v3/domains` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:00.251987 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fb92f02-4d00-474e-8215-133249852326 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:12:00.253027 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fb92f02-4d00-474e-8215-133249852326 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:00.255444 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fb92f02-4d00-474e-8215-133249852326 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:00.266884 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 842/1668] 10.222.0.77 () {58 vars in 1153 bytes} [Tue Aug 30 05:12:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 78 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:12:00.267899 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f65f9315-d2db-4c2d-8f0b-e7408bcd7201 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:00.295421 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f65f9315-d2db-4c2d-8f0b-e7408bcd7201 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:00.299008 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 827/1669] 10.222.0.77 () {58 vars in 1153 bytes} [Tue Aug 30 05:12:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:12:00.300050 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fb92f02-4d00-474e-8215-133249852326 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:00.303599 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2de31dfd-901a-412e-baf2-03a17aee1820 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:00.304766 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2de31dfd-901a-412e-baf2-03a17aee1820 admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:00.304766 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2de31dfd-901a-412e-baf2-03a17aee1820 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:00.304766 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2de31dfd-901a-412e-baf2-03a17aee1820 admin admin] PATH_INFO: `/v3/projects` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:00.306180 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2de31dfd-901a-412e-baf2-03a17aee1820 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:00.308118 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2de31dfd-901a-412e-baf2-03a17aee1820 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:00.335274 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fb92f02-4d00-474e-8215-133249852326 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:00.351373 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33e990bd-a1d1-4333-aa12-4173417acd0d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:00.352006 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33e990bd-a1d1-4333-aa12-4173417acd0d admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:00.352415 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33e990bd-a1d1-4333-aa12-4173417acd0d admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:00.352706 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33e990bd-a1d1-4333-aa12-4173417acd0d admin admin] PATH_INFO: `/v3/projects` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:00.353403 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33e990bd-a1d1-4333-aa12-4173417acd0d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:00.356282 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33e990bd-a1d1-4333-aa12-4173417acd0d admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:00.387306 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 843/1670] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:12:00 2022] POST /identity/v3/projects => generated 365 bytes in 117 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:12:00.388457 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2de31dfd-901a-412e-baf2-03a17aee1820 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:00.423308 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 828/1671] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:12:00 2022] POST /identity/v3/projects => generated 375 bytes in 124 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:12:00.424514 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33e990bd-a1d1-4333-aa12-4173417acd0d admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:00.544320 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2de31dfd-901a-412e-baf2-03a17aee1820 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:00.611655 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33e990bd-a1d1-4333-aa12-4173417acd0d admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:00.621758 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00e8e333-b4bf-4322-a4c5-da1004c0eb56 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:00.622923 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e8e333-b4bf-4322-a4c5-da1004c0eb56 admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:00.623474 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e8e333-b4bf-4322-a4c5-da1004c0eb56 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:00.624534 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e8e333-b4bf-4322-a4c5-da1004c0eb56 admin admin] PATH_INFO: `/v3/users` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:00.625705 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00e8e333-b4bf-4322-a4c5-da1004c0eb56 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:00.628388 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00e8e333-b4bf-4322-a4c5-da1004c0eb56 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:00.694673 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-969a5b55-e46e-4812-8e6d-6c450e2e389b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:00.695369 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-969a5b55-e46e-4812-8e6d-6c450e2e389b admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:00.695655 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-969a5b55-e46e-4812-8e6d-6c450e2e389b admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:00.695911 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-969a5b55-e46e-4812-8e6d-6c450e2e389b admin admin] PATH_INFO: `/v3/users` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:00.697367 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-969a5b55-e46e-4812-8e6d-6c450e2e389b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:00.699242 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-969a5b55-e46e-4812-8e6d-6c450e2e389b admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:00.732970 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 829/1672] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:12:00 2022] POST /identity/v3/users => generated 288 bytes in 306 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:12:00.741650 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c6cf25-b545-4186-8010-f1772b6d4a95 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:00.741845 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c6cf25-b545-4186-8010-f1772b6d4a95 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:00.742080 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c6cf25-b545-4186-8010-f1772b6d4a95 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:00.745498 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 844/1673] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:12:00 2022] POST /identity/v3/users => generated 283 bytes in 357 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:12:00.746398 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-969a5b55-e46e-4812-8e6d-6c450e2e389b admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:00.766625 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-969a5b55-e46e-4812-8e6d-6c450e2e389b admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:00.776962 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8e9dc64-67b6-4235-9f59-f3a306ea06af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:00.776962 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8e9dc64-67b6-4235-9f59-f3a306ea06af admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:00.777350 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8e9dc64-67b6-4235-9f59-f3a306ea06af admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:00.777573 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8e9dc64-67b6-4235-9f59-f3a306ea06af admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:00.778228 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8e9dc64-67b6-4235-9f59-f3a306ea06af admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:00.779942 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8e9dc64-67b6-4235-9f59-f3a306ea06af admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:00.807823 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 845/1674] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:12:00 2022] GET /identity/v3/roles => generated 1420 bytes in 62 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:12:00.808729 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8e9dc64-67b6-4235-9f59-f3a306ea06af admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:00.825740 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8e9dc64-67b6-4235-9f59-f3a306ea06af admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:00.828807 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-16c6cf25-b545-4186-8010-f1772b6d4a95 None None] MFA Rules not processed for user `d611a49b03554b448a09fe86604af02b`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:12:00.834532 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-16c6cf25-b545-4186-8010-f1772b6d4a95 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:00.839488 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bbcd567-124c-4185-8231-853ad4af2476 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:00.840212 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bbcd567-124c-4185-8231-853ad4af2476 admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:00.840544 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bbcd567-124c-4185-8231-853ad4af2476 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:00.840916 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bbcd567-124c-4185-8231-853ad4af2476 admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:00.845338 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bbcd567-124c-4185-8231-853ad4af2476 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:00.847123 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bbcd567-124c-4185-8231-853ad4af2476 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:00.860149 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 846/1675] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:12:00 2022] GET /identity/v3/roles => generated 1420 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:12:00.861032 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bbcd567-124c-4185-8231-853ad4af2476 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:00.893025 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bbcd567-124c-4185-8231-853ad4af2476 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:00.901259 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-109809f4-5d29-4416-a1d3-174dd74f1839 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:00.901981 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-109809f4-5d29-4416-a1d3-174dd74f1839 admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:00.909170 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-109809f4-5d29-4416-a1d3-174dd74f1839 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:00.909665 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-109809f4-5d29-4416-a1d3-174dd74f1839 admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:00.911499 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-109809f4-5d29-4416-a1d3-174dd74f1839 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:00.931955 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-109809f4-5d29-4416-a1d3-174dd74f1839 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:00.956880 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 847/1676] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:12:00 2022] GET /identity/v3/roles => generated 1420 bytes in 96 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:12:00.958638 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-109809f4-5d29-4416-a1d3-174dd74f1839 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:01.002322 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-109809f4-5d29-4416-a1d3-174dd74f1839 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:01.008015 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5835d809-eb32-40fd-a332-6a9a82ba1f55 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:01.008977 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5835d809-eb32-40fd-a332-6a9a82ba1f55 admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:01.009537 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5835d809-eb32-40fd-a332-6a9a82ba1f55 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:01.010100 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5835d809-eb32-40fd-a332-6a9a82ba1f55 admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:01.026159 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5835d809-eb32-40fd-a332-6a9a82ba1f55 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:01.029259 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5835d809-eb32-40fd-a332-6a9a82ba1f55 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:01.037986 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 830/1677] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:12:00 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 307 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:12:01.039479 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16c6cf25-b545-4186-8010-f1772b6d4a95 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:01.057307 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 848/1678] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:12:00 2022] GET /identity/v3/roles => generated 1420 bytes in 99 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:12:01.058781 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c782e58c-dca8-4547-a81c-77214291c8ea None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:01.059193 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c782e58c-dca8-4547-a81c-77214291c8ea None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:01.059643 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c782e58c-dca8-4547-a81c-77214291c8ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:01.076500 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16c6cf25-b545-4186-8010-f1772b6d4a95 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:01.079015 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:01.088375 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29ec2051-8249-404d-bc11-381c22689da4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:01.089106 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ec2051-8249-404d-bc11-381c22689da4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:01.089935 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ec2051-8249-404d-bc11-381c22689da4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:01.090213 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ec2051-8249-404d-bc11-381c22689da4 admin admin] PATH_INFO: `/v3/projects/60b191df1134402f83194ae40d623eb7/users/e904acf61724406381ad67dbb2f3ad06/roles/6f03d86980c34c36b44a8d6764ff4fda` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:01.100976 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-c782e58c-dca8-4547-a81c-77214291c8ea None None] MFA Rules not processed for user `d611a49b03554b448a09fe86604af02b`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:12:01.104754 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-c782e58c-dca8-4547-a81c-77214291c8ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:01.127728 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29ec2051-8249-404d-bc11-381c22689da4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=60b191df1134402f83194ae40d623eb7, user_id=e904acf61724406381ad67dbb2f3ad06, role_id=6f03d86980c34c36b44a8d6764ff4fda)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:01.131268 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29ec2051-8249-404d-bc11-381c22689da4 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:01.133302 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 849/1679] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:12:01 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 75 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:12:01.134316 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c782e58c-dca8-4547-a81c-77214291c8ea None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:01.162800 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:01.168318 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 831/1680] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:12:01 2022] PUT /identity/v3/projects/60b191df1134402f83194ae40d623eb7/users/e904acf61724406381ad67dbb2f3ad06/roles/6f03d86980c34c36b44a8d6764ff4fda => generated 0 bytes in 130 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:12:01.179399 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29cb7643-a76f-4c45-8f84-14c2ea4f51e9 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:01.179692 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29cb7643-a76f-4c45-8f84-14c2ea4f51e9 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:01.179971 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29cb7643-a76f-4c45-8f84-14c2ea4f51e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:01.188083 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c782e58c-dca8-4547-a81c-77214291c8ea None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:01.277067 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-29cb7643-a76f-4c45-8f84-14c2ea4f51e9 None None] MFA Rules not processed for user `e904acf61724406381ad67dbb2f3ad06`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:12:01.278222 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-29cb7643-a76f-4c45-8f84-14c2ea4f51e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:01.390513 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc682bd2-85ca-4a26-965d-5fcbf514e60e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:01.392845 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc682bd2-85ca-4a26-965d-5fcbf514e60e admin admin] REQUEST_METHOD: `PUT` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:01.393577 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc682bd2-85ca-4a26-965d-5fcbf514e60e admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:01.395551 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc682bd2-85ca-4a26-965d-5fcbf514e60e admin admin] PATH_INFO: `/v3/projects/35a86b84fe4f494f8bcfa11a100cb7eb/users/54fb3815e31349ff81f6565007e97055/roles/6f03d86980c34c36b44a8d6764ff4fda` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:01.452117 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc682bd2-85ca-4a26-965d-5fcbf514e60e admin admin] RBAC: Authorizing `identity:create_grant(project_id=35a86b84fe4f494f8bcfa11a100cb7eb, user_id=54fb3815e31349ff81f6565007e97055, role_id=6f03d86980c34c36b44a8d6764ff4fda)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:01.456670 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc682bd2-85ca-4a26-965d-5fcbf514e60e admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:01.498272 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 850/1681] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:12:01 2022] PUT /identity/v3/projects/35a86b84fe4f494f8bcfa11a100cb7eb/users/54fb3815e31349ff81f6565007e97055/roles/6f03d86980c34c36b44a8d6764ff4fda => generated 0 bytes in 364 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:12:01.500011 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d868dfc7-7316-4685-a28f-ffbb98a12148 None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:01.500394 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d868dfc7-7316-4685-a28f-ffbb98a12148 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:01.501754 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d868dfc7-7316-4685-a28f-ffbb98a12148 None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:01.508099 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 851/1682] 10.222.0.77 () {54 vars in 962 bytes} [Tue Aug 30 05:12:01 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:12:01.513591 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-474bd546-b95f-454d-bb15-1956f2887bcb None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:01.516289 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-474bd546-b95f-454d-bb15-1956f2887bcb None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:01.517946 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-474bd546-b95f-454d-bb15-1956f2887bcb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:01.536951 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 832/1683] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:12:01 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 356 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:12:01.537406 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b4bd7c-e8cc-43a9-86ad-b13a34d31fea None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:01.542248 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b4bd7c-e8cc-43a9-86ad-b13a34d31fea None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:01.542533 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b4bd7c-e8cc-43a9-86ad-b13a34d31fea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:01.592325 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-474bd546-b95f-454d-bb15-1956f2887bcb None None] MFA Rules not processed for user `54fb3815e31349ff81f6565007e97055`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:12:01.596361 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-474bd546-b95f-454d-bb15-1956f2887bcb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:01.668308 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-e8b4bd7c-e8cc-43a9-86ad-b13a34d31fea None None] MFA Rules not processed for user `886eb5e00bbf4ec292eb962b665b37d0`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:12:01.669570 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-e8b4bd7c-e8cc-43a9-86ad-b13a34d31fea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:01.792587 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 852/1684] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:12:01 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 280 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:12:01.793359 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-474bd546-b95f-454d-bb15-1956f2887bcb None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:01.914039 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 833/1685] 10.222.0.77 () {58 vars in 1059 bytes} [Tue Aug 30 05:12:01 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 379 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:12:01.917340 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8b4bd7c-e8cc-43a9-86ad-b13a34d31fea None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:01.953467 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-474bd546-b95f-454d-bb15-1956f2887bcb None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:01.973399 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e849a7c-3a08-4b7c-97d0-dbb7b2208721 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:01.974140 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e849a7c-3a08-4b7c-97d0-dbb7b2208721 admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:01.977834 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e849a7c-3a08-4b7c-97d0-dbb7b2208721 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:01.978241 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e849a7c-3a08-4b7c-97d0-dbb7b2208721 admin admin] PATH_INFO: `/v3/projects` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:01.980150 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e849a7c-3a08-4b7c-97d0-dbb7b2208721 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:01.989158 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e849a7c-3a08-4b7c-97d0-dbb7b2208721 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:02.055954 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:02.100549 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 853/1686] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:12:01 2022] POST /identity/v3/projects => generated 375 bytes in 304 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:12:02.100549 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e849a7c-3a08-4b7c-97d0-dbb7b2208721 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:02.114638 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8b4bd7c-e8cc-43a9-86ad-b13a34d31fea None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:02.149282 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:02.349705 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51236cf4-c063-442c-bb24-4acd013d0cff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:02.352757 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51236cf4-c063-442c-bb24-4acd013d0cff admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:02.355571 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51236cf4-c063-442c-bb24-4acd013d0cff admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:02.355859 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51236cf4-c063-442c-bb24-4acd013d0cff admin admin] PATH_INFO: `/v3/projects` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:02.361547 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51236cf4-c063-442c-bb24-4acd013d0cff admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:02.383984 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51236cf4-c063-442c-bb24-4acd013d0cff admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:02.387884 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e849a7c-3a08-4b7c-97d0-dbb7b2208721 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:02.401172 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7307ee69-8a9d-4c10-aa10-7a962d95b1c3 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '886eb5e00bbf4ec292eb962b665b37d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:02.403178 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7307ee69-8a9d-4c10-aa10-7a962d95b1c3 service nova] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:02.403455 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7307ee69-8a9d-4c10-aa10-7a962d95b1c3 service nova] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:02.404881 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7307ee69-8a9d-4c10-aa10-7a962d95b1c3 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:02.451313 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 834/1687] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:12:01 2022] POST /identity/v3/projects => generated 363 bytes in 536 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:12:02.452627 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51236cf4-c063-442c-bb24-4acd013d0cff admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:02.565199 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7307ee69-8a9d-4c10-aa10-7a962d95b1c3 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:02.567568 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7307ee69-8a9d-4c10-aa10-7a962d95b1c3 service nova] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:02.589577 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 854/1688] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:12:02 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 492 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:12:02.591010 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51be28f1-1a20-4918-ab99-13b000324236 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:02.591295 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51be28f1-1a20-4918-ab99-13b000324236 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:02.591564 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51be28f1-1a20-4918-ab99-13b000324236 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:02.607398 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51236cf4-c063-442c-bb24-4acd013d0cff admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:02.619069 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33c96dd9-f86f-4090-8302-fab96572aca1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:02.620414 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33c96dd9-f86f-4090-8302-fab96572aca1 admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:02.622676 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33c96dd9-f86f-4090-8302-fab96572aca1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:02.623195 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33c96dd9-f86f-4090-8302-fab96572aca1 admin admin] PATH_INFO: `/v3/users` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:02.625614 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33c96dd9-f86f-4090-8302-fab96572aca1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:02.630980 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33c96dd9-f86f-4090-8302-fab96572aca1 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:02.659645 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-51be28f1-1a20-4918-ab99-13b000324236 None None] MFA Rules not processed for user `a8f01d0042c84767899dc2b71bca4e12`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:12:02.661506 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-51be28f1-1a20-4918-ab99-13b000324236 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:02.709144 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 835/1689] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:12:02 2022] POST /identity/v3/users => generated 294 bytes in 257 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:12:02.713148 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33c96dd9-f86f-4090-8302-fab96572aca1 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:02.759579 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33c96dd9-f86f-4090-8302-fab96572aca1 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:02.766550 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aafdbae-6152-4638-a5db-fd061ecf6eab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:02.767295 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aafdbae-6152-4638-a5db-fd061ecf6eab admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:02.767707 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aafdbae-6152-4638-a5db-fd061ecf6eab admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:02.768088 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aafdbae-6152-4638-a5db-fd061ecf6eab admin admin] PATH_INFO: `/v3/users` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:02.769216 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aafdbae-6152-4638-a5db-fd061ecf6eab admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:02.772784 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aafdbae-6152-4638-a5db-fd061ecf6eab admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:02.808439 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 855/1690] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:12:02 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 218 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:12:02.814714 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 836/1691] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:12:02 2022] POST /identity/v3/users => generated 288 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:12:02.817254 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51be28f1-1a20-4918-ab99-13b000324236 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:02.819466 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aafdbae-6152-4638-a5db-fd061ecf6eab admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:02.834928 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aafdbae-6152-4638-a5db-fd061ecf6eab admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:02.842068 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51be28f1-1a20-4918-ab99-13b000324236 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:02.848083 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-558514a1-94f5-4170-a527-e8b161a357c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:02.850595 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-558514a1-94f5-4170-a527-e8b161a357c9 admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:02.851001 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c872718-6ddb-4253-a633-f0618c0acd54 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:02.851534 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-558514a1-94f5-4170-a527-e8b161a357c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:02.851651 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c872718-6ddb-4253-a633-f0618c0acd54 admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:02.851920 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c872718-6ddb-4253-a633-f0618c0acd54 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:02.852205 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c872718-6ddb-4253-a633-f0618c0acd54 admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:02.852782 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-558514a1-94f5-4170-a527-e8b161a357c9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:02.852890 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c872718-6ddb-4253-a633-f0618c0acd54 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:02.854564 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c872718-6ddb-4253-a633-f0618c0acd54 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:02.855078 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-558514a1-94f5-4170-a527-e8b161a357c9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:02.857555 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-558514a1-94f5-4170-a527-e8b161a357c9 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:02.883853 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 856/1692] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:12:02 2022] GET /identity/v3/roles => generated 1420 bytes in 72 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:12:02.885180 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c872718-6ddb-4253-a633-f0618c0acd54 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:02.895314 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 837/1693] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:12:02 2022] GET /identity/v3/roles => generated 1420 bytes in 75 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 05:12:02.899349 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38a07217-7a21-4d27-9151-930488cfb315 None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:02.900172 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38a07217-7a21-4d27-9151-930488cfb315 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:02.900735 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38a07217-7a21-4d27-9151-930488cfb315 None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:02.903338 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 838/1694] 10.222.0.77 () {54 vars in 962 bytes} [Tue Aug 30 05:12:02 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:12:02.904840 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38a07217-7a21-4d27-9151-930488cfb315 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:02.905822 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c872718-6ddb-4253-a633-f0618c0acd54 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:02.928716 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ac0bd62-49a2-4d8e-a4a2-e4404e9e2657 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:02.931308 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac0bd62-49a2-4d8e-a4a2-e4404e9e2657 admin admin] REQUEST_METHOD: `PUT` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:02.933510 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38a07217-7a21-4d27-9151-930488cfb315 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:02.933648 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac0bd62-49a2-4d8e-a4a2-e4404e9e2657 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:02.934943 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac0bd62-49a2-4d8e-a4a2-e4404e9e2657 admin admin] PATH_INFO: `/v3/projects/b8d0da5bcfa140c9a9e5e69d1b2ad94c/users/c37a7697ded84613843c630b4b0ac6a3/roles/9d9d5d2c270146488249b7ed943280db` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:02.941883 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8297b9b1-9519-4f77-af77-b2cd410b6d63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:02.944073 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8297b9b1-9519-4f77-af77-b2cd410b6d63 admin admin] REQUEST_METHOD: `PUT` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:02.944558 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8297b9b1-9519-4f77-af77-b2cd410b6d63 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:02.945451 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8297b9b1-9519-4f77-af77-b2cd410b6d63 admin admin] PATH_INFO: `/v3/projects/5202bce64d254549bcaa3e12464f4fbf/users/820dca7d92a34df893ab1d08177a070f/roles/9d9d5d2c270146488249b7ed943280db` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:02.973656 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:02.981259 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8297b9b1-9519-4f77-af77-b2cd410b6d63 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5202bce64d254549bcaa3e12464f4fbf, user_id=820dca7d92a34df893ab1d08177a070f, role_id=9d9d5d2c270146488249b7ed943280db)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:02.984667 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8297b9b1-9519-4f77-af77-b2cd410b6d63 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:03.005179 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ac0bd62-49a2-4d8e-a4a2-e4404e9e2657 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b8d0da5bcfa140c9a9e5e69d1b2ad94c, user_id=c37a7697ded84613843c630b4b0ac6a3, role_id=9d9d5d2c270146488249b7ed943280db)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:03.008638 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 839/1695] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:12:02 2022] PUT /identity/v3/projects/5202bce64d254549bcaa3e12464f4fbf/users/820dca7d92a34df893ab1d08177a070f/roles/9d9d5d2c270146488249b7ed943280db => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:12:03.010268 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc0799fe-4fb5-4fab-84e8-b3a59b8c122a None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:03.010719 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc0799fe-4fb5-4fab-84e8-b3a59b8c122a None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:03.011548 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc0799fe-4fb5-4fab-84e8-b3a59b8c122a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:03.020123 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ac0bd62-49a2-4d8e-a4a2-e4404e9e2657 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:03.056584 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-dc0799fe-4fb5-4fab-84e8-b3a59b8c122a None None] MFA Rules not processed for user `886eb5e00bbf4ec292eb962b665b37d0`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:12:03.058913 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-dc0799fe-4fb5-4fab-84e8-b3a59b8c122a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:03.064679 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 857/1696] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:12:02 2022] PUT /identity/v3/projects/b8d0da5bcfa140c9a9e5e69d1b2ad94c/users/c37a7697ded84613843c630b4b0ac6a3/roles/9d9d5d2c270146488249b7ed943280db => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:12:03.065612 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ac0bd62-49a2-4d8e-a4a2-e4404e9e2657 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:03.119085 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:03.198208 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 840/1697] 10.222.0.77 () {58 vars in 1059 bytes} [Tue Aug 30 05:12:03 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:12:03.199459 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc0799fe-4fb5-4fab-84e8-b3a59b8c122a None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:03.239544 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ac0bd62-49a2-4d8e-a4a2-e4404e9e2657 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:03.251120 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98536eb4-d20d-44a9-9c9c-29e61bb3d7bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:03.251752 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98536eb4-d20d-44a9-9c9c-29e61bb3d7bd admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:03.252035 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98536eb4-d20d-44a9-9c9c-29e61bb3d7bd admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:03.252306 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98536eb4-d20d-44a9-9c9c-29e61bb3d7bd admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:03.252958 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98536eb4-d20d-44a9-9c9c-29e61bb3d7bd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:03.255287 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98536eb4-d20d-44a9-9c9c-29e61bb3d7bd admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:03.285079 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 858/1698] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:12:03 2022] GET /identity/v3/roles => generated 1420 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:12:03.286029 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98536eb4-d20d-44a9-9c9c-29e61bb3d7bd admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:03.325986 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98536eb4-d20d-44a9-9c9c-29e61bb3d7bd admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:03.343164 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6450155-9ae9-473d-946e-eb9b1af5c197 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '886eb5e00bbf4ec292eb962b665b37d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:03.343704 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6450155-9ae9-473d-946e-eb9b1af5c197 service nova] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:03.343984 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6450155-9ae9-473d-946e-eb9b1af5c197 service nova] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:03.344258 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6450155-9ae9-473d-946e-eb9b1af5c197 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:03.385364 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc0799fe-4fb5-4fab-84e8-b3a59b8c122a None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:03.415220 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6700d0c5-6449-449d-acb2-78b56e119fb1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:03.416805 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6700d0c5-6449-449d-acb2-78b56e119fb1 admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:03.416805 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6700d0c5-6449-449d-acb2-78b56e119fb1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:03.416805 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6700d0c5-6449-449d-acb2-78b56e119fb1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:03.421774 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6700d0c5-6449-449d-acb2-78b56e119fb1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:03.431417 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6700d0c5-6449-449d-acb2-78b56e119fb1 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:03.467846 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 841/1699] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:12:03 2022] GET /identity/v3/roles => generated 1420 bytes in 269 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:12:03.468699 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6700d0c5-6449-449d-acb2-78b56e119fb1 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:03.493616 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6700d0c5-6449-449d-acb2-78b56e119fb1 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:03.512992 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1376042e-8804-444e-b97b-ada276a23da3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:03.513669 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1376042e-8804-444e-b97b-ada276a23da3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:03.513952 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1376042e-8804-444e-b97b-ada276a23da3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:03.514242 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1376042e-8804-444e-b97b-ada276a23da3 admin admin] PATH_INFO: `/v3/projects/5202bce64d254549bcaa3e12464f4fbf/users/820dca7d92a34df893ab1d08177a070f/roles/9d9d5d2c270146488249b7ed943280db` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:03.520378 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6450155-9ae9-473d-946e-eb9b1af5c197 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:03.522998 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1376042e-8804-444e-b97b-ada276a23da3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5202bce64d254549bcaa3e12464f4fbf, user_id=820dca7d92a34df893ab1d08177a070f, role_id=9d9d5d2c270146488249b7ed943280db)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:03.525443 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6450155-9ae9-473d-946e-eb9b1af5c197 service nova] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:03.525677 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1376042e-8804-444e-b97b-ada276a23da3 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:03.550043 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 842/1700] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:12:03 2022] PUT /identity/v3/projects/5202bce64d254549bcaa3e12464f4fbf/users/820dca7d92a34df893ab1d08177a070f/roles/9d9d5d2c270146488249b7ed943280db => generated 0 bytes in 82 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:12:03.551475 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1376042e-8804-444e-b97b-ada276a23da3 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:03.561830 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 859/1701] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:12:03 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 276 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:12:03.563242 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff24576b-8494-4a48-b4eb-881bf9954212 None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:03.563533 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff24576b-8494-4a48-b4eb-881bf9954212 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:03.563811 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff24576b-8494-4a48-b4eb-881bf9954212 None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:03.572007 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 860/1702] 10.222.0.77 () {54 vars in 912 bytes} [Tue Aug 30 05:12:03 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:12:03.573335 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c44d27e1-7c1d-42e5-9346-9efcd70e9aac None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:03.573627 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c44d27e1-7c1d-42e5-9346-9efcd70e9aac None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:03.573905 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c44d27e1-7c1d-42e5-9346-9efcd70e9aac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:03.650933 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-c44d27e1-7c1d-42e5-9346-9efcd70e9aac None None] MFA Rules not processed for user `820dca7d92a34df893ab1d08177a070f`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:12:03.652187 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-c44d27e1-7c1d-42e5-9346-9efcd70e9aac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:03.701151 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1376042e-8804-444e-b97b-ada276a23da3 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:03.708395 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2dc33e0-7e9b-4eef-a2e9-e19860be27bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:03.709161 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2dc33e0-7e9b-4eef-a2e9-e19860be27bf admin admin] REQUEST_METHOD: `PUT` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:03.709516 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2dc33e0-7e9b-4eef-a2e9-e19860be27bf admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:03.709852 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2dc33e0-7e9b-4eef-a2e9-e19860be27bf admin admin] PATH_INFO: `/v3/projects/b8d0da5bcfa140c9a9e5e69d1b2ad94c/users/c37a7697ded84613843c630b4b0ac6a3/roles/9d9d5d2c270146488249b7ed943280db` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:03.713517 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2dc33e0-7e9b-4eef-a2e9-e19860be27bf admin admin] RBAC: Authorizing `identity:create_grant(project_id=b8d0da5bcfa140c9a9e5e69d1b2ad94c, user_id=c37a7697ded84613843c630b4b0ac6a3, role_id=9d9d5d2c270146488249b7ed943280db)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:03.718156 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2dc33e0-7e9b-4eef-a2e9-e19860be27bf admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:03.745364 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 843/1703] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:12:03 2022] PUT /identity/v3/projects/b8d0da5bcfa140c9a9e5e69d1b2ad94c/users/c37a7697ded84613843c630b4b0ac6a3/roles/9d9d5d2c270146488249b7ed943280db => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:12:03.747115 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-039fbac2-2ea1-4d09-b63f-3e483d4f1a81 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:03.748722 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-039fbac2-2ea1-4d09-b63f-3e483d4f1a81 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:03.749007 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-039fbac2-2ea1-4d09-b63f-3e483d4f1a81 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:03.783106 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-039fbac2-2ea1-4d09-b63f-3e483d4f1a81 None None] MFA Rules not processed for user `886eb5e00bbf4ec292eb962b665b37d0`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:12:03.784531 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-039fbac2-2ea1-4d09-b63f-3e483d4f1a81 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:03.921337 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 861/1704] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:12:03 2022] POST /identity/v3/auth/tokens => generated 3307 bytes in 345 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:12:03.921337 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b8378b-be6d-467e-a4a0-4915ac22fb69 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:03.921337 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b8378b-be6d-467e-a4a0-4915ac22fb69 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:03.922421 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b8378b-be6d-467e-a4a0-4915ac22fb69 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:03.998157 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:03.999326 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:04.056976 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-e9b8378b-be6d-467e-a4a0-4915ac22fb69 None None] MFA Rules not processed for user `c37a7697ded84613843c630b4b0ac6a3`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:12:04.105110 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-e9b8378b-be6d-467e-a4a0-4915ac22fb69 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:04.156665 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 844/1705] 10.222.0.77 () {58 vars in 1009 bytes} [Tue Aug 30 05:12:03 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 411 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:12:04.174407 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-039fbac2-2ea1-4d09-b63f-3e483d4f1a81 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:04.176675 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-039fbac2-2ea1-4d09-b63f-3e483d4f1a81 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:04.421512 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 862/1706] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:12:03 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 503 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:12:04.423597 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9b8378b-be6d-467e-a4a0-4915ac22fb69 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:04.426944 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-e9b8378b-be6d-467e-a4a0-4915ac22fb69 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:04.498156 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-039fbac2-2ea1-4d09-b63f-3e483d4f1a81 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:04.507794 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b75b1e60-2086-445d-aeb6-f267cb49bb63 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a002ac0bf240a891ac7304178d3f0e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:04.511807 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b75b1e60-2086-445d-aeb6-f267cb49bb63 service glance] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:04.512304 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b75b1e60-2086-445d-aeb6-f267cb49bb63 service glance] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:04.512648 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b75b1e60-2086-445d-aeb6-f267cb49bb63 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:04.547502 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b75b1e60-2086-445d-aeb6-f267cb49bb63 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:04.551808 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b75b1e60-2086-445d-aeb6-f267cb49bb63 service glance] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:04.604355 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 845/1707] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:12:04 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 431 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 05:12:04.644062 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9b8378b-be6d-467e-a4a0-4915ac22fb69 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:04.657043 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a74cce2-5e47-4d7c-8393-277965ec4d58 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '019a79b2affe478b8a30d1c538d0415c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:04.657675 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a74cce2-5e47-4d7c-8393-277965ec4d58 service cinder] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:04.658101 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a74cce2-5e47-4d7c-8393-277965ec4d58 service cinder] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:04.658378 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a74cce2-5e47-4d7c-8393-277965ec4d58 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:04.718905 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b75b1e60-2086-445d-aeb6-f267cb49bb63 service glance] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:04.721451 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-b75b1e60-2086-445d-aeb6-f267cb49bb63 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:04.745363 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b75b1e60-2086-445d-aeb6-f267cb49bb63 service glance] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:04.758549 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a2d4b6e-2ec0-4a64-9d4a-ea30b5de3e92 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '019a79b2affe478b8a30d1c538d0415c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:04.760168 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a2d4b6e-2ec0-4a64-9d4a-ea30b5de3e92 service cinder] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:04.760691 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a2d4b6e-2ec0-4a64-9d4a-ea30b5de3e92 service cinder] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:04.761129 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a2d4b6e-2ec0-4a64-9d4a-ea30b5de3e92 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:04.899029 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a74cce2-5e47-4d7c-8393-277965ec4d58 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:04.900822 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a74cce2-5e47-4d7c-8393-277965ec4d58 service cinder] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:04.911158 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a2d4b6e-2ec0-4a64-9d4a-ea30b5de3e92 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:04.922664 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a2d4b6e-2ec0-4a64-9d4a-ea30b5de3e92 service cinder] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:04.925102 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 863/1708] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:12:04 2022] GET /identity/v3/auth/tokens => generated 3307 bytes in 503 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:12:04.961471 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 846/1709] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:12:04 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 241 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:12:05.211412 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1166fc62-3d35-4b81-9ecb-af959c476bf5 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:05.211701 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1166fc62-3d35-4b81-9ecb-af959c476bf5 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:05.211975 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1166fc62-3d35-4b81-9ecb-af959c476bf5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:05.252412 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a2d4b6e-2ec0-4a64-9d4a-ea30b5de3e92 service cinder] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:05.263743 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a2d4b6e-2ec0-4a64-9d4a-ea30b5de3e92 service cinder] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:05.280680 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6450f130-6fe0-40b8-ab09-e18b002cfdcd service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '019a79b2affe478b8a30d1c538d0415c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:05.282463 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6450f130-6fe0-40b8-ab09-e18b002cfdcd service cinder] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:05.282885 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6450f130-6fe0-40b8-ab09-e18b002cfdcd service cinder] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:05.283415 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6450f130-6fe0-40b8-ab09-e18b002cfdcd service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:05.336183 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-1166fc62-3d35-4b81-9ecb-af959c476bf5 None None] MFA Rules not processed for user `2fe451987a8c49c5b5fc0ef1c9c30ab5`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:12:05.339501 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-1166fc62-3d35-4b81-9ecb-af959c476bf5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:05.398529 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6450f130-6fe0-40b8-ab09-e18b002cfdcd service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:05.402317 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6450f130-6fe0-40b8-ab09-e18b002cfdcd service cinder] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:05.407420 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:05.427207 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 847/1710] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:12:05 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 175 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 05:12:05.430025 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccbbe08a-16f3-43f9-8d40-9c889495aa0b None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:05.430466 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccbbe08a-16f3-43f9-8d40-9c889495aa0b None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:05.430964 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccbbe08a-16f3-43f9-8d40-9c889495aa0b None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:05.443256 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 848/1711] 10.222.0.77 () {54 vars in 912 bytes} [Tue Aug 30 05:12:05 2022] GET /identity => generated 270 bytes in 11 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:12:05.444464 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:05.446648 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21424329-9cb1-4ed5-a98b-3ec8a22a8130 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:05.447193 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21424329-9cb1-4ed5-a98b-3ec8a22a8130 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:05.449656 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21424329-9cb1-4ed5-a98b-3ec8a22a8130 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:05.561663 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 864/1712] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:12:05 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 352 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:12:05.562892 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-21424329-9cb1-4ed5-a98b-3ec8a22a8130 None None] MFA Rules not processed for user `c33c45f4770d4151858e1d897d09af69`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:12:05.564437 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-21424329-9cb1-4ed5-a98b-3ec8a22a8130 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:05.588730 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1166fc62-3d35-4b81-9ecb-af959c476bf5 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:05.612220 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1166fc62-3d35-4b81-9ecb-af959c476bf5 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:05.625456 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ebee8db-541b-4271-98c0-b5767ee417ec service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '886eb5e00bbf4ec292eb962b665b37d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:05.626745 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ebee8db-541b-4271-98c0-b5767ee417ec service nova] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:05.627213 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ebee8db-541b-4271-98c0-b5767ee417ec service nova] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:05.627635 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ebee8db-541b-4271-98c0-b5767ee417ec service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:05.660765 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ebee8db-541b-4271-98c0-b5767ee417ec service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:05.662791 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ebee8db-541b-4271-98c0-b5767ee417ec service nova] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:05.705548 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 865/1713] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:12:05 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 05:12:05.774237 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 849/1714] 10.222.0.77 () {58 vars in 1009 bytes} [Tue Aug 30 05:12:05 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 330 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:12:05.790723 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ebee8db-541b-4271-98c0-b5767ee417ec service nova] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:05.794381 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-8ebee8db-541b-4271-98c0-b5767ee417ec service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:05.820504 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ebee8db-541b-4271-98c0-b5767ee417ec service nova] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:05.834347 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9682e0a-6282-48de-8bfe-0bf9c29eb881 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:05.835161 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9682e0a-6282-48de-8bfe-0bf9c29eb881 service neutron] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:05.835363 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9682e0a-6282-48de-8bfe-0bf9c29eb881 service neutron] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:05.837057 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9682e0a-6282-48de-8bfe-0bf9c29eb881 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:05.865480 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9682e0a-6282-48de-8bfe-0bf9c29eb881 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:05.868725 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9682e0a-6282-48de-8bfe-0bf9c29eb881 service neutron] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:05.925724 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 866/1715] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:12:05 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 134 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:12:06.217749 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9c6a3dd-4882-4f29-b4ae-b85e93634ff2 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:06.218262 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9c6a3dd-4882-4f29-b4ae-b85e93634ff2 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:06.218620 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9c6a3dd-4882-4f29-b4ae-b85e93634ff2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:06.301789 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-e9c6a3dd-4882-4f29-b4ae-b85e93634ff2 None None] MFA Rules not processed for user `31cc6aa564be4c569e6fa89813cb9329`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:12:06.303189 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-e9c6a3dd-4882-4f29-b4ae-b85e93634ff2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:06.310263 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9682e0a-6282-48de-8bfe-0bf9c29eb881 service neutron] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:06.312118 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-e9682e0a-6282-48de-8bfe-0bf9c29eb881 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:06.505244 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9682e0a-6282-48de-8bfe-0bf9c29eb881 service neutron] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:06.509891 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 850/1716] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:12:06 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 293 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:12:06.521160 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52b292ed-8a4a-410d-8476-7461afb32d79 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:06.521907 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52b292ed-8a4a-410d-8476-7461afb32d79 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:06.523531 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52b292ed-8a4a-410d-8476-7461afb32d79 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:06.532724 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72a0394d-ec52-46f1-aee9-a2ff993da118 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:06.533481 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a0394d-ec52-46f1-aee9-a2ff993da118 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:06.541099 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a0394d-ec52-46f1-aee9-a2ff993da118 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:06.541757 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a0394d-ec52-46f1-aee9-a2ff993da118 admin admin] PATH_INFO: `/v3/users/34204351339944f1995ce340ca36409b` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:06.548148 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72a0394d-ec52-46f1-aee9-a2ff993da118 admin admin] RBAC: Authorizing `identity:delete_user(user_id=34204351339944f1995ce340ca36409b)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:06.553549 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72a0394d-ec52-46f1-aee9-a2ff993da118 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:06.594346 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-52b292ed-8a4a-410d-8476-7461afb32d79 None None] MFA Rules not processed for user `31cc6aa564be4c569e6fa89813cb9329`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:12:06.609625 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-52b292ed-8a4a-410d-8476-7461afb32d79 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:06.623574 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 851/1717] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:12:06 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:12:06.634043 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52b292ed-8a4a-410d-8476-7461afb32d79 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:06.641722 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-72a0394d-ec52-46f1-aee9-a2ff993da118 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '34204351339944f1995ce340ca36409b'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:06.655020 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-72a0394d-ec52-46f1-aee9-a2ff993da118 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '34204351339944f1995ce340ca36409b'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:06.669051 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52b292ed-8a4a-410d-8476-7461afb32d79 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:06.677709 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-72a0394d-ec52-46f1-aee9-a2ff993da118 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '34204351339944f1995ce340ca36409b'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:06.688065 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-72a0394d-ec52-46f1-aee9-a2ff993da118 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '34204351339944f1995ce340ca36409b'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:06.689087 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08c01310-f40a-49cf-89d6-7b19943982e3 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '886eb5e00bbf4ec292eb962b665b37d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:06.689495 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-72a0394d-ec52-46f1-aee9-a2ff993da118 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '34204351339944f1995ce340ca36409b'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:06.690058 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08c01310-f40a-49cf-89d6-7b19943982e3 service nova] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:06.692135 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08c01310-f40a-49cf-89d6-7b19943982e3 service nova] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:06.693581 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08c01310-f40a-49cf-89d6-7b19943982e3 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:06.697864 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 867/1718] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:12:06 2022] DELETE /identity/v3/users/34204351339944f1995ce340ca36409b => generated 0 bytes in 388 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:12:06.701714 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-08c01310-f40a-49cf-89d6-7b19943982e3 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:06.702621 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b3cb987-1f9c-4907-8ae5-923419846451 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:06.703212 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b3cb987-1f9c-4907-8ae5-923419846451 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:06.703676 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b3cb987-1f9c-4907-8ae5-923419846451 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:06.723140 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08c01310-f40a-49cf-89d6-7b19943982e3 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:06.727243 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08c01310-f40a-49cf-89d6-7b19943982e3 service nova] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:06.745027 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-8b3cb987-1f9c-4907-8ae5-923419846451 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:12:06.747202 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-8b3cb987-1f9c-4907-8ae5-923419846451 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:06.759988 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 852/1719] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:12:06 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 126 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:12:06.776419 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:06.830444 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 868/1720] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:12:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:12:06.841230 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08c01310-f40a-49cf-89d6-7b19943982e3 service nova] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:06.842665 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-08c01310-f40a-49cf-89d6-7b19943982e3 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:06.865299 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08c01310-f40a-49cf-89d6-7b19943982e3 service nova] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:06.875261 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a3c19de-eee9-4049-91fc-0e1300dea221 None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:06.876156 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a3c19de-eee9-4049-91fc-0e1300dea221 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:06.876535 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a3c19de-eee9-4049-91fc-0e1300dea221 None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:06.878134 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 869/1721] 10.222.0.77 () {54 vars in 912 bytes} [Tue Aug 30 05:12:06 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 05:12:06.891546 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e84e93d-8032-4595-83a7-70b04a314ac9 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:06.891921 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e84e93d-8032-4595-83a7-70b04a314ac9 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:06.892349 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e84e93d-8032-4595-83a7-70b04a314ac9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:06.935808 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-1e84e93d-8032-4595-83a7-70b04a314ac9 None None] MFA Rules not processed for user `886eb5e00bbf4ec292eb962b665b37d0`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:12:06.939828 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-1e84e93d-8032-4595-83a7-70b04a314ac9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:07.005492 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69c7b672-c96b-455b-bfb9-210eea7a1f96 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:07.008528 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c7b672-c96b-455b-bfb9-210eea7a1f96 service neutron] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:07.009264 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c7b672-c96b-455b-bfb9-210eea7a1f96 service neutron] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:07.009264 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c7b672-c96b-455b-bfb9-210eea7a1f96 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:07.034631 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69c7b672-c96b-455b-bfb9-210eea7a1f96 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:07.044352 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69c7b672-c96b-455b-bfb9-210eea7a1f96 service neutron] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:07.119527 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 853/1722] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:12:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 276 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:12:07.158384 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 870/1723] 10.222.0.77 () {58 vars in 1009 bytes} [Tue Aug 30 05:12:06 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 268 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 05:12:07.165782 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69c7b672-c96b-455b-bfb9-210eea7a1f96 service neutron] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:07.167801 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-69c7b672-c96b-455b-bfb9-210eea7a1f96 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:07.179143 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:07.295624 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e84e93d-8032-4595-83a7-70b04a314ac9 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:07.297135 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-1e84e93d-8032-4595-83a7-70b04a314ac9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:07.298013 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69c7b672-c96b-455b-bfb9-210eea7a1f96 service neutron] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:07.311438 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-467a90ac-ee5e-45cf-acc4-d6b98480966b service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a002ac0bf240a891ac7304178d3f0e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:07.312130 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-467a90ac-ee5e-45cf-acc4-d6b98480966b service glance] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:07.315520 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-467a90ac-ee5e-45cf-acc4-d6b98480966b service glance] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:07.315826 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-467a90ac-ee5e-45cf-acc4-d6b98480966b service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:07.366359 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-467a90ac-ee5e-45cf-acc4-d6b98480966b service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:07.372634 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-467a90ac-ee5e-45cf-acc4-d6b98480966b service glance] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:07.431652 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 854/1724] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:12:07 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 266 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:12:07.466776 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e84e93d-8032-4595-83a7-70b04a314ac9 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:07.486865 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d62fd991-7f68-48a9-94e7-2069ae7b0408 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1972959402704b89925001cd8d90cb0e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:07.487675 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d62fd991-7f68-48a9-94e7-2069ae7b0408 service placement] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:07.488117 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d62fd991-7f68-48a9-94e7-2069ae7b0408 service placement] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:07.488260 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d62fd991-7f68-48a9-94e7-2069ae7b0408 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:07.510866 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-d62fd991-7f68-48a9-94e7-2069ae7b0408 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:07.538599 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d62fd991-7f68-48a9-94e7-2069ae7b0408 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:07.540271 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d62fd991-7f68-48a9-94e7-2069ae7b0408 service placement] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:07.577714 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 871/1725] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:12:07 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 284 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:12:08.297354 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3ea57ec-4d43-4bcd-9182-062d1da98792 None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:08.297832 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3ea57ec-4d43-4bcd-9182-062d1da98792 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:08.297923 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3ea57ec-4d43-4bcd-9182-062d1da98792 None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:08.303059 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 855/1726] 10.222.0.77 () {54 vars in 912 bytes} [Tue Aug 30 05:12:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:12:08.305781 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ada0e24-9d3b-4df3-9216-38e45b01e141 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:08.306242 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ada0e24-9d3b-4df3-9216-38e45b01e141 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:08.306334 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ada0e24-9d3b-4df3-9216-38e45b01e141 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:08.359623 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-3ada0e24-9d3b-4df3-9216-38e45b01e141 None None] MFA Rules not processed for user `c33c45f4770d4151858e1d897d09af69`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:12:08.364464 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-3ada0e24-9d3b-4df3-9216-38e45b01e141 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:08.380049 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 872/1727] 10.222.0.77 () {58 vars in 1009 bytes} [Tue Aug 30 05:12:08 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 75 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:12:08.449247 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3ea57ec-4d43-4bcd-9182-062d1da98792 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:08.451085 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-c3ea57ec-4d43-4bcd-9182-062d1da98792 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:08.465644 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ada0e24-9d3b-4df3-9216-38e45b01e141 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:08.477363 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ada0e24-9d3b-4df3-9216-38e45b01e141 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:08.490403 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3ea57ec-4d43-4bcd-9182-062d1da98792 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:08.508423 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f132af01-649d-4f92-994d-544837e6487e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:08.509290 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f132af01-649d-4f92-994d-544837e6487e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:08.509722 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f132af01-649d-4f92-994d-544837e6487e admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:08.510541 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f132af01-649d-4f92-994d-544837e6487e admin admin] PATH_INFO: `/v3/projects/558c2bea073546cabaffbf54e1d85a14` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:08.511937 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f132af01-649d-4f92-994d-544837e6487e admin admin] RBAC: Authorizing `identity:delete_project(project_id=558c2bea073546cabaffbf54e1d85a14)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:08.514277 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f132af01-649d-4f92-994d-544837e6487e admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:08.535929 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0e0ef47-f379-4f71-a399-0d0b1ab372f2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:08.540246 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0e0ef47-f379-4f71-a399-0d0b1ab372f2 service neutron] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:08.544235 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0e0ef47-f379-4f71-a399-0d0b1ab372f2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:08.545625 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0e0ef47-f379-4f71-a399-0d0b1ab372f2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:08.585188 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0e0ef47-f379-4f71-a399-0d0b1ab372f2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:08.589142 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0e0ef47-f379-4f71-a399-0d0b1ab372f2 service neutron] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:08.598673 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-f132af01-649d-4f92-994d-544837e6487e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '558c2bea073546cabaffbf54e1d85a14', 'request_id': 'req-f132af01-649d-4f92-994d-544837e6487e'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:08.616234 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-f132af01-649d-4f92-994d-544837e6487e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '558c2bea073546cabaffbf54e1d85a14', 'request_id': 'req-f132af01-649d-4f92-994d-544837e6487e'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:08.628796 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-f132af01-649d-4f92-994d-544837e6487e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '558c2bea073546cabaffbf54e1d85a14', 'request_id': 'req-f132af01-649d-4f92-994d-544837e6487e'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:08.660758 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 873/1728] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:12:08 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 197 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 05:12:08.663967 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-f132af01-649d-4f92-994d-544837e6487e admin admin] The token cache is being invalidate because project 558c2bea073546cabaffbf54e1d85a14 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81624) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:12:08.664489 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-f132af01-649d-4f92-994d-544837e6487e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:08.665600 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-f132af01-649d-4f92-994d-544837e6487e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:08.669722 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 856/1729] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:12:08 2022] DELETE /identity/v3/projects/558c2bea073546cabaffbf54e1d85a14 => generated 0 bytes in 221 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:12:08.673929 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0e0ef47-f379-4f71-a399-0d0b1ab372f2 service neutron] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:08.690157 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0e0ef47-f379-4f71-a399-0d0b1ab372f2 service neutron] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:08.732278 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:08.863403 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71f892c0-2d68-4524-b619-a327f18923d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:08.864134 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71f892c0-2d68-4524-b619-a327f18923d5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:08.864417 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71f892c0-2d68-4524-b619-a327f18923d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:08.866509 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71f892c0-2d68-4524-b619-a327f18923d5 admin admin] PATH_INFO: `/v3/users/f1c7e121df9f4e588790c586ceb6e7a8` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:08.869247 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71f892c0-2d68-4524-b619-a327f18923d5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f1c7e121df9f4e588790c586ceb6e7a8)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:08.875852 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71f892c0-2d68-4524-b619-a327f18923d5 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:08.995538 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-71f892c0-2d68-4524-b619-a327f18923d5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f1c7e121df9f4e588790c586ceb6e7a8'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:09.034001 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-71f892c0-2d68-4524-b619-a327f18923d5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f1c7e121df9f4e588790c586ceb6e7a8'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:09.058788 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-71f892c0-2d68-4524-b619-a327f18923d5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f1c7e121df9f4e588790c586ceb6e7a8'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:09.081397 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-71f892c0-2d68-4524-b619-a327f18923d5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f1c7e121df9f4e588790c586ceb6e7a8'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:09.085319 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-71f892c0-2d68-4524-b619-a327f18923d5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f1c7e121df9f4e588790c586ceb6e7a8'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:09.093643 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 874/1730] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:12:08 2022] DELETE /identity/v3/users/f1c7e121df9f4e588790c586ceb6e7a8 => generated 0 bytes in 420 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:12:09.917445 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec984cab-b6f2-4de5-8bf6-49a3ab93b6ef None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:09.917445 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec984cab-b6f2-4de5-8bf6-49a3ab93b6ef None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:09.917445 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec984cab-b6f2-4de5-8bf6-49a3ab93b6ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:09.965685 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-ec984cab-b6f2-4de5-8bf6-49a3ab93b6ef None None] MFA Rules not processed for user `31cc6aa564be4c569e6fa89813cb9329`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:12:09.972304 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-ec984cab-b6f2-4de5-8bf6-49a3ab93b6ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:10.169969 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 857/1731] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:12:09 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 255 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:12:10.307924 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf26d969-6a37-46b3-a2eb-3849831f0b2b None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:10.308477 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf26d969-6a37-46b3-a2eb-3849831f0b2b None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:10.308549 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf26d969-6a37-46b3-a2eb-3849831f0b2b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:10.353709 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-cf26d969-6a37-46b3-a2eb-3849831f0b2b None None] MFA Rules not processed for user `a8f01d0042c84767899dc2b71bca4e12`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:12:10.360508 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-cf26d969-6a37-46b3-a2eb-3849831f0b2b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:10.586776 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 875/1732] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:12:10 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 280 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:12:10.612346 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b36d6dc6-86d8-46d7-800e-1a572c16df47 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:10.612886 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b36d6dc6-86d8-46d7-800e-1a572c16df47 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:10.614135 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b36d6dc6-86d8-46d7-800e-1a572c16df47 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:10.694204 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-b36d6dc6-86d8-46d7-800e-1a572c16df47 None None] MFA Rules not processed for user `2fe451987a8c49c5b5fc0ef1c9c30ab5`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:12:10.697342 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-b36d6dc6-86d8-46d7-800e-1a572c16df47 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:10.724827 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf26d969-6a37-46b3-a2eb-3849831f0b2b None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:10.726643 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-cf26d969-6a37-46b3-a2eb-3849831f0b2b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:10.869196 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:10.941259 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:10.954907 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 858/1733] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:12:10 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 336 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:12:10.957789 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b36d6dc6-86d8-46d7-800e-1a572c16df47 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:10.962595 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-b36d6dc6-86d8-46d7-800e-1a572c16df47 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:10.987087 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf26d969-6a37-46b3-a2eb-3849831f0b2b None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:11.006166 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2145c4fa-7690-45d8-bca6-6bfde548d96f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:11.007093 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2145c4fa-7690-45d8-bca6-6bfde548d96f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:11.007405 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2145c4fa-7690-45d8-bca6-6bfde548d96f admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:11.007688 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2145c4fa-7690-45d8-bca6-6bfde548d96f admin admin] PATH_INFO: `/v3/projects/35e2d52f0b9c45dfa6235c9ed9bbc4b6` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:11.016465 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2145c4fa-7690-45d8-bca6-6bfde548d96f admin admin] RBAC: Authorizing `identity:delete_project(project_id=35e2d52f0b9c45dfa6235c9ed9bbc4b6)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:11.018683 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2145c4fa-7690-45d8-bca6-6bfde548d96f admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:11.134791 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b36d6dc6-86d8-46d7-800e-1a572c16df47 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:11.163247 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-2145c4fa-7690-45d8-bca6-6bfde548d96f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '35e2d52f0b9c45dfa6235c9ed9bbc4b6', 'request_id': 'req-2145c4fa-7690-45d8-bca6-6bfde548d96f'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:11.182625 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-2145c4fa-7690-45d8-bca6-6bfde548d96f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '35e2d52f0b9c45dfa6235c9ed9bbc4b6', 'request_id': 'req-2145c4fa-7690-45d8-bca6-6bfde548d96f'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:11.194163 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-2145c4fa-7690-45d8-bca6-6bfde548d96f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '35e2d52f0b9c45dfa6235c9ed9bbc4b6', 'request_id': 'req-2145c4fa-7690-45d8-bca6-6bfde548d96f'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:11.233335 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-2145c4fa-7690-45d8-bca6-6bfde548d96f admin admin] The token cache is being invalidate because project 35e2d52f0b9c45dfa6235c9ed9bbc4b6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81625) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:12:11.233637 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-2145c4fa-7690-45d8-bca6-6bfde548d96f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:11.236271 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-2145c4fa-7690-45d8-bca6-6bfde548d96f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:11.239239 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 876/1734] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:12:10 2022] DELETE /identity/v3/projects/35e2d52f0b9c45dfa6235c9ed9bbc4b6 => generated 0 bytes in 515 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 05:12:11.244445 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad4d289b-43a7-485f-bb9b-4db6c0f55cf8 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:11.244739 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad4d289b-43a7-485f-bb9b-4db6c0f55cf8 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:11.245016 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad4d289b-43a7-485f-bb9b-4db6c0f55cf8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:11.275730 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d61cb9c-b06d-4d78-8333-170c59cb6652 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '019a79b2affe478b8a30d1c538d0415c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:11.276949 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d61cb9c-b06d-4d78-8333-170c59cb6652 service cinder] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:11.276949 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d61cb9c-b06d-4d78-8333-170c59cb6652 service cinder] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:11.277078 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d61cb9c-b06d-4d78-8333-170c59cb6652 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:11.284435 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-4d61cb9c-b06d-4d78-8333-170c59cb6652 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:11.316685 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-ad4d289b-43a7-485f-bb9b-4db6c0f55cf8 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:12:11.318054 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-ad4d289b-43a7-485f-bb9b-4db6c0f55cf8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:11.451771 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d61cb9c-b06d-4d78-8333-170c59cb6652 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:11.457946 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d61cb9c-b06d-4d78-8333-170c59cb6652 service cinder] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:11.523843 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 859/1735] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:12:10 2022] GET /identity/v3/auth/tokens => generated 3319 bytes in 568 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:12:11.597176 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 877/1736] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:12:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 353 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:12:11.606805 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096af56d-8847-427a-ad39-0b2dec820ccb None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:11.607247 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096af56d-8847-427a-ad39-0b2dec820ccb None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:11.607587 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096af56d-8847-427a-ad39-0b2dec820ccb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:11.656236 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-096af56d-8847-427a-ad39-0b2dec820ccb None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:12:11.657732 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-096af56d-8847-427a-ad39-0b2dec820ccb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:11.685828 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 860/1737] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:12:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:12:11.686725 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad4d289b-43a7-485f-bb9b-4db6c0f55cf8 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:11.697651 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad4d289b-43a7-485f-bb9b-4db6c0f55cf8 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:11.703105 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f95921ed-1f19-43d3-80d8-1acde59fb8ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:11.704167 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f95921ed-1f19-43d3-80d8-1acde59fb8ae admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:11.704874 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f95921ed-1f19-43d3-80d8-1acde59fb8ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:11.705374 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f95921ed-1f19-43d3-80d8-1acde59fb8ae admin admin] PATH_INFO: `/v3/domains` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:11.706330 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f95921ed-1f19-43d3-80d8-1acde59fb8ae admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:12:11.706937 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f95921ed-1f19-43d3-80d8-1acde59fb8ae admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:11.708811 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f95921ed-1f19-43d3-80d8-1acde59fb8ae admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:11.729456 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 878/1738] 10.222.0.77 () {58 vars in 1153 bytes} [Tue Aug 30 05:12:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 05:12:11.733483 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-096af56d-8847-427a-ad39-0b2dec820ccb None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:11.750065 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-096af56d-8847-427a-ad39-0b2dec820ccb None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:11.759530 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e970538e-20b3-47dd-8d59-ca58397ad9ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:11.760204 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e970538e-20b3-47dd-8d59-ca58397ad9ee admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:11.760503 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e970538e-20b3-47dd-8d59-ca58397ad9ee admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:11.760785 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e970538e-20b3-47dd-8d59-ca58397ad9ee admin admin] PATH_INFO: `/v3/projects` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:11.761571 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e970538e-20b3-47dd-8d59-ca58397ad9ee admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:11.767846 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e970538e-20b3-47dd-8d59-ca58397ad9ee admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:11.804083 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 861/1739] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:12:11 2022] POST /identity/v3/projects => generated 375 bytes in 71 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:12:11.808829 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f95921ed-1f19-43d3-80d8-1acde59fb8ae admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:11.823755 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f95921ed-1f19-43d3-80d8-1acde59fb8ae admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:12.033646 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74077ef3-ef8e-40f1-8389-89100f72cb18 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:12.037772 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74077ef3-ef8e-40f1-8389-89100f72cb18 admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:12.039884 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74077ef3-ef8e-40f1-8389-89100f72cb18 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:12.041614 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74077ef3-ef8e-40f1-8389-89100f72cb18 admin admin] PATH_INFO: `/v3/users` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:12.048465 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74077ef3-ef8e-40f1-8389-89100f72cb18 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:12.052784 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74077ef3-ef8e-40f1-8389-89100f72cb18 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:12.133026 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:12.153445 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 879/1740] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:12:11 2022] POST /identity/v3/users => generated 288 bytes in 343 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:12:12.154643 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e970538e-20b3-47dd-8d59-ca58397ad9ee admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:12.178026 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e970538e-20b3-47dd-8d59-ca58397ad9ee admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:12.193644 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a8de874-08d1-4a16-863b-77480763d60b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:12.194272 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a8de874-08d1-4a16-863b-77480763d60b admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:12.194549 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a8de874-08d1-4a16-863b-77480763d60b admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:12.194837 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a8de874-08d1-4a16-863b-77480763d60b admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:12.195506 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a8de874-08d1-4a16-863b-77480763d60b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:12.209905 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a8de874-08d1-4a16-863b-77480763d60b admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:12.266985 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 862/1741] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:12:12 2022] GET /identity/v3/roles => generated 1420 bytes in 110 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:12:12.267787 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74077ef3-ef8e-40f1-8389-89100f72cb18 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:12.278948 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74077ef3-ef8e-40f1-8389-89100f72cb18 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:12.303531 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26d1f397-729e-4272-96e5-fa70573ee944 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:12.306751 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26d1f397-729e-4272-96e5-fa70573ee944 admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:12.309005 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26d1f397-729e-4272-96e5-fa70573ee944 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:12.310185 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26d1f397-729e-4272-96e5-fa70573ee944 admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:12.314067 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a8de874-08d1-4a16-863b-77480763d60b admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:12.316248 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-5a8de874-08d1-4a16-863b-77480763d60b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:12.326930 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26d1f397-729e-4272-96e5-fa70573ee944 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:12.335273 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26d1f397-729e-4272-96e5-fa70573ee944 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:12.357457 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a8de874-08d1-4a16-863b-77480763d60b admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:12.383906 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d7ba0c8-f0d2-49f9-8755-bf2ab0db246e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:12.387329 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d7ba0c8-f0d2-49f9-8755-bf2ab0db246e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:12.387615 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d7ba0c8-f0d2-49f9-8755-bf2ab0db246e admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:12.387889 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d7ba0c8-f0d2-49f9-8755-bf2ab0db246e admin admin] PATH_INFO: `/v3/users/54fb3815e31349ff81f6565007e97055` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:12.404732 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d7ba0c8-f0d2-49f9-8755-bf2ab0db246e admin admin] RBAC: Authorizing `identity:delete_user(user_id=54fb3815e31349ff81f6565007e97055)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:12.407394 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 880/1742] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:12:12 2022] GET /identity/v3/roles => generated 1420 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:12:12.409297 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26d1f397-729e-4272-96e5-fa70573ee944 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:12.413869 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: WARNING py.warnings [None req-9d7ba0c8-f0d2-49f9-8755-bf2ab0db246e admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 30 05:12:12.413869 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: warnings.warn(msg) Aug 30 05:12:12.413869 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: Aug 30 05:12:12.415196 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d7ba0c8-f0d2-49f9-8755-bf2ab0db246e admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:12.458516 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:12.487173 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26d1f397-729e-4272-96e5-fa70573ee944 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:12.492617 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-885b3803-9a4a-4b45-8c1b-f3801aba4f90 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:12.493768 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-885b3803-9a4a-4b45-8c1b-f3801aba4f90 admin admin] REQUEST_METHOD: `PUT` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:12.494192 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-885b3803-9a4a-4b45-8c1b-f3801aba4f90 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:12.494615 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-885b3803-9a4a-4b45-8c1b-f3801aba4f90 admin admin] PATH_INFO: `/v3/projects/7534ac52c2524faaabb38edb8d476695/users/5e0615e5403b4e47937426a43291d704/roles/6f03d86980c34c36b44a8d6764ff4fda` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:12.544004 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-885b3803-9a4a-4b45-8c1b-f3801aba4f90 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7534ac52c2524faaabb38edb8d476695, user_id=5e0615e5403b4e47937426a43291d704, role_id=6f03d86980c34c36b44a8d6764ff4fda)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:12.553029 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-885b3803-9a4a-4b45-8c1b-f3801aba4f90 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:12.595481 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 881/1743] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:12:12 2022] PUT /identity/v3/projects/7534ac52c2524faaabb38edb8d476695/users/5e0615e5403b4e47937426a43291d704/roles/6f03d86980c34c36b44a8d6764ff4fda => generated 0 bytes in 188 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:12:12.607026 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a04418-ed3f-4a62-919e-067c43c3dd18 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:12.607760 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a04418-ed3f-4a62-919e-067c43c3dd18 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:12.608241 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a04418-ed3f-4a62-919e-067c43c3dd18 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:12.759170 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-76a04418-ed3f-4a62-919e-067c43c3dd18 None None] MFA Rules not processed for user `5e0615e5403b4e47937426a43291d704`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:12:12.766008 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-76a04418-ed3f-4a62-919e-067c43c3dd18 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:12.886133 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-9d7ba0c8-f0d2-49f9-8755-bf2ab0db246e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '54fb3815e31349ff81f6565007e97055'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:12.887084 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-9d7ba0c8-f0d2-49f9-8755-bf2ab0db246e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '54fb3815e31349ff81f6565007e97055'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:12.888962 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-9d7ba0c8-f0d2-49f9-8755-bf2ab0db246e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '54fb3815e31349ff81f6565007e97055'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:12.974937 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-9d7ba0c8-f0d2-49f9-8755-bf2ab0db246e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '54fb3815e31349ff81f6565007e97055'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:13.008537 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-9d7ba0c8-f0d2-49f9-8755-bf2ab0db246e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '54fb3815e31349ff81f6565007e97055'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:13.055238 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 863/1744] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:12:12 2022] DELETE /identity/v3/users/54fb3815e31349ff81f6565007e97055 => generated 0 bytes in 741 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:12:13.058033 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a3b6fd-523b-4270-b6de-0182b79e9a6b None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:13.059226 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a3b6fd-523b-4270-b6de-0182b79e9a6b None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:13.059618 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a3b6fd-523b-4270-b6de-0182b79e9a6b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:13.071667 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 882/1745] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:12:12 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 466 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 05:12:13.075255 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca0193b-67ce-486c-84aa-1a4e7daa8e36 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:13.075669 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca0193b-67ce-486c-84aa-1a4e7daa8e36 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:13.076559 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca0193b-67ce-486c-84aa-1a4e7daa8e36 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:13.128598 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-5ca0193b-67ce-486c-84aa-1a4e7daa8e36 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:12:13.131095 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-5ca0193b-67ce-486c-84aa-1a4e7daa8e36 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:13.208248 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-d4a3b6fd-523b-4270-b6de-0182b79e9a6b None None] MFA Rules not processed for user `fe51dbec91624ed1ab00b441e29a3dd5`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:12:13.210604 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-d4a3b6fd-523b-4270-b6de-0182b79e9a6b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:13.287674 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 883/1746] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:12:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 215 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:12:13.289538 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ca0193b-67ce-486c-84aa-1a4e7daa8e36 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:13.293919 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-5ca0193b-67ce-486c-84aa-1a4e7daa8e36 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:13.334557 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ca0193b-67ce-486c-84aa-1a4e7daa8e36 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:13.351771 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93d35ecb-83df-4157-b890-c6f47c52bfbe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:13.353692 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d35ecb-83df-4157-b890-c6f47c52bfbe admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:13.354499 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d35ecb-83df-4157-b890-c6f47c52bfbe admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:13.355720 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d35ecb-83df-4157-b890-c6f47c52bfbe admin admin] PATH_INFO: `/v3/projects` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:13.357048 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93d35ecb-83df-4157-b890-c6f47c52bfbe admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:13.364595 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93d35ecb-83df-4157-b890-c6f47c52bfbe admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:13.438170 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 864/1747] 10.222.0.77 () {58 vars in 1009 bytes} [Tue Aug 30 05:12:13 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 382 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:12:13.441891 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4a3b6fd-523b-4270-b6de-0182b79e9a6b None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:13.455869 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-d4a3b6fd-523b-4270-b6de-0182b79e9a6b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:13.482111 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 884/1748] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:12:13 2022] POST /identity/v3/projects => generated 371 bytes in 193 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:12:13.486588 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93d35ecb-83df-4157-b890-c6f47c52bfbe admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:13.543269 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:13.628342 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4a3b6fd-523b-4270-b6de-0182b79e9a6b None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:13.638491 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:13.642921 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93d35ecb-83df-4157-b890-c6f47c52bfbe admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:13.662747 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11bc22b3-6057-42cb-977f-2bd056d079d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:13.663599 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11bc22b3-6057-42cb-977f-2bd056d079d5 admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:13.663993 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11bc22b3-6057-42cb-977f-2bd056d079d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:13.664401 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11bc22b3-6057-42cb-977f-2bd056d079d5 admin admin] PATH_INFO: `/v3/users` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:13.665182 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11bc22b3-6057-42cb-977f-2bd056d079d5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:13.672907 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11bc22b3-6057-42cb-977f-2bd056d079d5 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:13.728230 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d450721c-4c95-4f53-8511-ade1507b91a5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:13.729544 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d450721c-4c95-4f53-8511-ade1507b91a5 service neutron] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:13.730057 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d450721c-4c95-4f53-8511-ade1507b91a5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:13.730481 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d450721c-4c95-4f53-8511-ade1507b91a5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:13.752438 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 885/1749] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:12:13 2022] POST /identity/v3/users => generated 292 bytes in 266 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:12:13.754839 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11bc22b3-6057-42cb-977f-2bd056d079d5 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:13.758593 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d450721c-4c95-4f53-8511-ade1507b91a5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:13.767311 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d450721c-4c95-4f53-8511-ade1507b91a5 service neutron] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:13.781258 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11bc22b3-6057-42cb-977f-2bd056d079d5 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:13.792129 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61d041a2-4363-4b35-b99c-f78726ae93ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:13.792784 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d041a2-4363-4b35-b99c-f78726ae93ab admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:13.793070 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d041a2-4363-4b35-b99c-f78726ae93ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:13.793467 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d041a2-4363-4b35-b99c-f78726ae93ab admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:13.794018 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61d041a2-4363-4b35-b99c-f78726ae93ab admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:13.801203 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 865/1750] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:12:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 363 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:12:13.801616 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61d041a2-4363-4b35-b99c-f78726ae93ab admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:13.802051 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d450721c-4c95-4f53-8511-ade1507b91a5 service neutron] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:13.808033 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-d450721c-4c95-4f53-8511-ade1507b91a5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:13.824553 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 886/1751] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:12:13 2022] GET /identity/v3/roles => generated 1420 bytes in 71 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 05:12:13.828211 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61d041a2-4363-4b35-b99c-f78726ae93ab admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:13.838498 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61d041a2-4363-4b35-b99c-f78726ae93ab admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:13.843517 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22f6026e-a5bd-491f-ba98-ae33f394d92e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:13.845035 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22f6026e-a5bd-491f-ba98-ae33f394d92e admin admin] REQUEST_METHOD: `PUT` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:13.845035 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22f6026e-a5bd-491f-ba98-ae33f394d92e admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:13.845035 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22f6026e-a5bd-491f-ba98-ae33f394d92e admin admin] PATH_INFO: `/v3/projects/d4e7dd205fc948898d3a0004183a0f06/users/80188bcd72894a7797d995c6cfdf54e3/roles/9d9d5d2c270146488249b7ed943280db` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:13.881153 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22f6026e-a5bd-491f-ba98-ae33f394d92e admin admin] RBAC: Authorizing `identity:create_grant(project_id=d4e7dd205fc948898d3a0004183a0f06, user_id=80188bcd72894a7797d995c6cfdf54e3, role_id=9d9d5d2c270146488249b7ed943280db)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:13.883929 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22f6026e-a5bd-491f-ba98-ae33f394d92e admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:13.917364 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 887/1752] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:12:13 2022] PUT /identity/v3/projects/d4e7dd205fc948898d3a0004183a0f06/users/80188bcd72894a7797d995c6cfdf54e3/roles/9d9d5d2c270146488249b7ed943280db => generated 0 bytes in 87 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 05:12:13.918666 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22f6026e-a5bd-491f-ba98-ae33f394d92e admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:14.038440 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d450721c-4c95-4f53-8511-ade1507b91a5 service neutron] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:14.088267 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22f6026e-a5bd-491f-ba98-ae33f394d92e admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:14.104175 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93166a63-ed53-42ed-864a-e251fb44f0dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:14.105075 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93166a63-ed53-42ed-864a-e251fb44f0dc admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:14.107824 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93166a63-ed53-42ed-864a-e251fb44f0dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:14.107824 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93166a63-ed53-42ed-864a-e251fb44f0dc admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:14.107824 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93166a63-ed53-42ed-864a-e251fb44f0dc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:14.110565 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93166a63-ed53-42ed-864a-e251fb44f0dc admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:14.151290 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 888/1753] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:12:13 2022] GET /identity/v3/roles => generated 1420 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 05:12:14.152350 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93166a63-ed53-42ed-864a-e251fb44f0dc admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:14.164718 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ad20baa-24a2-412d-86a7-f699294d2992 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a7f7afc3d44ec788c05d005e52a1b8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:14.165598 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93166a63-ed53-42ed-864a-e251fb44f0dc admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:14.166964 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ad20baa-24a2-412d-86a7-f699294d2992 service swift] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:14.167318 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ad20baa-24a2-412d-86a7-f699294d2992 service swift] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:14.167930 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ad20baa-24a2-412d-86a7-f699294d2992 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:14.171172 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8e91569-b1d8-4eec-83a5-8602afb7504c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:14.171931 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8e91569-b1d8-4eec-83a5-8602afb7504c admin admin] REQUEST_METHOD: `PUT` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:14.172216 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8e91569-b1d8-4eec-83a5-8602afb7504c admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:14.172488 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8e91569-b1d8-4eec-83a5-8602afb7504c admin admin] PATH_INFO: `/v3/projects/d4e7dd205fc948898d3a0004183a0f06/users/80188bcd72894a7797d995c6cfdf54e3/roles/9d9d5d2c270146488249b7ed943280db` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:14.174506 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8e91569-b1d8-4eec-83a5-8602afb7504c admin admin] RBAC: Authorizing `identity:create_grant(project_id=d4e7dd205fc948898d3a0004183a0f06, user_id=80188bcd72894a7797d995c6cfdf54e3, role_id=9d9d5d2c270146488249b7ed943280db)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:14.178430 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8e91569-b1d8-4eec-83a5-8602afb7504c admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:14.214573 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 889/1754] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:12:14 2022] PUT /identity/v3/projects/d4e7dd205fc948898d3a0004183a0f06/users/80188bcd72894a7797d995c6cfdf54e3/roles/9d9d5d2c270146488249b7ed943280db => generated 0 bytes in 63 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:12:14.227585 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ee46d96-57bc-4561-ae65-235142040f54 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:14.227877 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ee46d96-57bc-4561-ae65-235142040f54 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:14.230308 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ee46d96-57bc-4561-ae65-235142040f54 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:14.279717 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:14.295816 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-9ee46d96-57bc-4561-ae65-235142040f54 None None] MFA Rules not processed for user `80188bcd72894a7797d995c6cfdf54e3`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:12:14.297050 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-9ee46d96-57bc-4561-ae65-235142040f54 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:14.348095 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ad20baa-24a2-412d-86a7-f699294d2992 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:14.350702 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ad20baa-24a2-412d-86a7-f699294d2992 service swift] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:14.371498 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 866/1755] 10.222.0.77 () {60 vars in 1384 bytes} [Tue Aug 30 05:12:13 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 570 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 30 05:12:14.399835 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ad20baa-24a2-412d-86a7-f699294d2992 service swift] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:14.405211 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-3ad20baa-24a2-412d-86a7-f699294d2992 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:14.442005 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ad20baa-24a2-412d-86a7-f699294d2992 service swift] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:14.470002 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:14.493792 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 890/1756] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:12:14 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 267 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:12:14.526338 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ee46d96-57bc-4561-ae65-235142040f54 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:14.528116 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-9ee46d96-57bc-4561-ae65-235142040f54 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:14.575688 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec635f5b-1471-4acf-a2c3-f6516a2c232e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:14.576513 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec635f5b-1471-4acf-a2c3-f6516a2c232e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:14.577869 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec635f5b-1471-4acf-a2c3-f6516a2c232e admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:14.578184 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec635f5b-1471-4acf-a2c3-f6516a2c232e admin admin] PATH_INFO: `/v3/projects/35a86b84fe4f494f8bcfa11a100cb7eb` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:14.580523 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec635f5b-1471-4acf-a2c3-f6516a2c232e admin admin] RBAC: Authorizing `identity:delete_project(project_id=35a86b84fe4f494f8bcfa11a100cb7eb)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:14.600880 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec635f5b-1471-4acf-a2c3-f6516a2c232e admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:14.668173 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ee46d96-57bc-4561-ae65-235142040f54 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:14.685822 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8332e466-87cc-4502-a606-a4d4a9be21a1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '019a79b2affe478b8a30d1c538d0415c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:14.687319 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8332e466-87cc-4502-a606-a4d4a9be21a1 service cinder] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:14.692265 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8332e466-87cc-4502-a606-a4d4a9be21a1 service cinder] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:14.692265 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8332e466-87cc-4502-a606-a4d4a9be21a1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:14.785770 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-ec635f5b-1471-4acf-a2c3-f6516a2c232e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '35a86b84fe4f494f8bcfa11a100cb7eb', 'request_id': 'req-ec635f5b-1471-4acf-a2c3-f6516a2c232e'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:14.799992 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-ec635f5b-1471-4acf-a2c3-f6516a2c232e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '35a86b84fe4f494f8bcfa11a100cb7eb', 'request_id': 'req-ec635f5b-1471-4acf-a2c3-f6516a2c232e'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:14.810251 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-ec635f5b-1471-4acf-a2c3-f6516a2c232e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '35a86b84fe4f494f8bcfa11a100cb7eb', 'request_id': 'req-ec635f5b-1471-4acf-a2c3-f6516a2c232e'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:14.841790 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8332e466-87cc-4502-a606-a4d4a9be21a1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:14.843582 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8332e466-87cc-4502-a606-a4d4a9be21a1 service cinder] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:14.845221 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-ec635f5b-1471-4acf-a2c3-f6516a2c232e admin admin] The token cache is being invalidate because project 35a86b84fe4f494f8bcfa11a100cb7eb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81624) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:12:14.845792 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-ec635f5b-1471-4acf-a2c3-f6516a2c232e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:14.846873 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-ec635f5b-1471-4acf-a2c3-f6516a2c232e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:14.849004 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 867/1757] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:12:14 2022] DELETE /identity/v3/projects/35a86b84fe4f494f8bcfa11a100cb7eb => generated 0 bytes in 450 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:12:14.853468 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec635f5b-1471-4acf-a2c3-f6516a2c232e admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:14.855020 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-ec635f5b-1471-4acf-a2c3-f6516a2c232e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:14.855290 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 891/1758] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:12:14 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 330 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 05:12:14.989708 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec635f5b-1471-4acf-a2c3-f6516a2c232e admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:15.007016 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d6991f6-78a4-4727-9a33-18c6cbce3bfe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:15.010572 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d6991f6-78a4-4727-9a33-18c6cbce3bfe admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:15.012257 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d6991f6-78a4-4727-9a33-18c6cbce3bfe admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:15.012832 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d6991f6-78a4-4727-9a33-18c6cbce3bfe admin admin] PATH_INFO: `/v3/users/820dca7d92a34df893ab1d08177a070f` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:15.014163 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d6991f6-78a4-4727-9a33-18c6cbce3bfe admin admin] RBAC: Authorizing `identity:delete_user(user_id=820dca7d92a34df893ab1d08177a070f)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:15.017276 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d6991f6-78a4-4727-9a33-18c6cbce3bfe admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:15.029444 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:15.088246 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-0d6991f6-78a4-4727-9a33-18c6cbce3bfe admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '820dca7d92a34df893ab1d08177a070f'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:15.089207 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-0d6991f6-78a4-4727-9a33-18c6cbce3bfe admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '820dca7d92a34df893ab1d08177a070f'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:15.089987 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-0d6991f6-78a4-4727-9a33-18c6cbce3bfe admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '820dca7d92a34df893ab1d08177a070f'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:15.111933 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-0d6991f6-78a4-4727-9a33-18c6cbce3bfe admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '820dca7d92a34df893ab1d08177a070f'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:15.127958 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-0d6991f6-78a4-4727-9a33-18c6cbce3bfe admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '820dca7d92a34df893ab1d08177a070f'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:15.150039 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 868/1759] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:12:14 2022] DELETE /identity/v3/users/820dca7d92a34df893ab1d08177a070f => generated 0 bytes in 297 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:12:15.549501 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8332e466-87cc-4502-a606-a4d4a9be21a1 service cinder] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:15.567046 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8332e466-87cc-4502-a606-a4d4a9be21a1 service cinder] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:15.674987 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5178b34d-a730-4d3c-a6ae-c058ca9e7cfa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:15.676188 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5178b34d-a730-4d3c-a6ae-c058ca9e7cfa admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:15.676468 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5178b34d-a730-4d3c-a6ae-c058ca9e7cfa admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:15.676740 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5178b34d-a730-4d3c-a6ae-c058ca9e7cfa admin admin] PATH_INFO: `/v3/projects/5202bce64d254549bcaa3e12464f4fbf` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:15.678198 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5178b34d-a730-4d3c-a6ae-c058ca9e7cfa admin admin] RBAC: Authorizing `identity:delete_project(project_id=5202bce64d254549bcaa3e12464f4fbf)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:15.680396 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5178b34d-a730-4d3c-a6ae-c058ca9e7cfa admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:15.744647 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-5178b34d-a730-4d3c-a6ae-c058ca9e7cfa admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5202bce64d254549bcaa3e12464f4fbf', 'request_id': 'req-5178b34d-a730-4d3c-a6ae-c058ca9e7cfa'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:15.746650 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=96,keystone:DELETE=4 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:15.756372 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-5178b34d-a730-4d3c-a6ae-c058ca9e7cfa admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5202bce64d254549bcaa3e12464f4fbf', 'request_id': 'req-5178b34d-a730-4d3c-a6ae-c058ca9e7cfa'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:15.766191 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-5178b34d-a730-4d3c-a6ae-c058ca9e7cfa admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5202bce64d254549bcaa3e12464f4fbf', 'request_id': 'req-5178b34d-a730-4d3c-a6ae-c058ca9e7cfa'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:15.778936 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-5178b34d-a730-4d3c-a6ae-c058ca9e7cfa admin admin] The token cache is being invalidate because project 5202bce64d254549bcaa3e12464f4fbf was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81625) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:12:15.779578 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-5178b34d-a730-4d3c-a6ae-c058ca9e7cfa admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:15.780437 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-5178b34d-a730-4d3c-a6ae-c058ca9e7cfa admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:15.782515 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 892/1760] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:12:15 2022] DELETE /identity/v3/projects/5202bce64d254549bcaa3e12464f4fbf => generated 0 bytes in 233 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:12:15.787705 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc4052d7-e549-4da6-b48b-4aa25f883bb5 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:15.788208 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc4052d7-e549-4da6-b48b-4aa25f883bb5 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:15.788283 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc4052d7-e549-4da6-b48b-4aa25f883bb5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:15.824020 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-dc4052d7-e549-4da6-b48b-4aa25f883bb5 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:12:15.824020 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-dc4052d7-e549-4da6-b48b-4aa25f883bb5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:15.914878 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 869/1761] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:12:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:12:15.916894 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66375844-5a8a-464e-9d1b-b89c9d517556 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:15.917180 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66375844-5a8a-464e-9d1b-b89c9d517556 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:15.917458 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66375844-5a8a-464e-9d1b-b89c9d517556 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:15.957192 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-66375844-5a8a-464e-9d1b-b89c9d517556 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:12:15.958486 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-66375844-5a8a-464e-9d1b-b89c9d517556 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:15.973330 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 893/1762] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:12:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 58 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 05:12:15.977779 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc4052d7-e549-4da6-b48b-4aa25f883bb5 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:15.990038 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc4052d7-e549-4da6-b48b-4aa25f883bb5 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:16.007104 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1f4deeb-a74b-4b85-9d9c-9d850afe5a85 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:16.009639 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f4deeb-a74b-4b85-9d9c-9d850afe5a85 admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:16.010368 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f4deeb-a74b-4b85-9d9c-9d850afe5a85 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:16.010717 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f4deeb-a74b-4b85-9d9c-9d850afe5a85 admin admin] PATH_INFO: `/v3/domains` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:16.012348 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1f4deeb-a74b-4b85-9d9c-9d850afe5a85 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:12:16.013143 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1f4deeb-a74b-4b85-9d9c-9d850afe5a85 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:16.014986 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1f4deeb-a74b-4b85-9d9c-9d850afe5a85 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:16.062107 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 870/1763] 10.222.0.77 () {58 vars in 1153 bytes} [Tue Aug 30 05:12:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 79 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:12:16.063859 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66375844-5a8a-464e-9d1b-b89c9d517556 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:16.078286 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66375844-5a8a-464e-9d1b-b89c9d517556 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:16.085171 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c92468f8-568d-4f11-9bb7-22cbd826508c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:16.086328 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c92468f8-568d-4f11-9bb7-22cbd826508c admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:16.086619 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c92468f8-568d-4f11-9bb7-22cbd826508c admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:16.086925 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c92468f8-568d-4f11-9bb7-22cbd826508c admin admin] PATH_INFO: `/v3/projects` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:16.087753 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c92468f8-568d-4f11-9bb7-22cbd826508c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:16.092422 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c92468f8-568d-4f11-9bb7-22cbd826508c admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:16.157911 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 894/1764] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:12:16 2022] POST /identity/v3/projects => generated 387 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 05:12:16.163078 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1f4deeb-a74b-4b85-9d9c-9d850afe5a85 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:16.194949 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1f4deeb-a74b-4b85-9d9c-9d850afe5a85 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:16.340158 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fcec5e2-bda1-4cf1-8c6e-976ca2dabba1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:16.340780 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fcec5e2-bda1-4cf1-8c6e-976ca2dabba1 admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:16.341058 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fcec5e2-bda1-4cf1-8c6e-976ca2dabba1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:16.341327 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fcec5e2-bda1-4cf1-8c6e-976ca2dabba1 admin admin] PATH_INFO: `/v3/users` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:16.341956 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fcec5e2-bda1-4cf1-8c6e-976ca2dabba1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:16.344407 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fcec5e2-bda1-4cf1-8c6e-976ca2dabba1 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:16.375567 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 871/1765] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:12:16 2022] POST /identity/v3/users => generated 294 bytes in 210 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:12:16.376974 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c92468f8-568d-4f11-9bb7-22cbd826508c admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:16.388139 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c92468f8-568d-4f11-9bb7-22cbd826508c admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:16.411878 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62bcda18-d17a-4c1a-aef5-e7bceebebb96 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:16.412568 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62bcda18-d17a-4c1a-aef5-e7bceebebb96 admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:16.412853 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62bcda18-d17a-4c1a-aef5-e7bceebebb96 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:16.413129 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62bcda18-d17a-4c1a-aef5-e7bceebebb96 admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:16.413781 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62bcda18-d17a-4c1a-aef5-e7bceebebb96 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:16.415625 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62bcda18-d17a-4c1a-aef5-e7bceebebb96 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:16.431938 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 895/1766] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:12:16 2022] GET /identity/v3/roles => generated 1420 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:12:16.435877 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fcec5e2-bda1-4cf1-8c6e-976ca2dabba1 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:16.442559 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=86,keystone:DELETE=10,keystone:INSERT=4 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:16.452308 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fcec5e2-bda1-4cf1-8c6e-976ca2dabba1 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:16.465303 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07868bb0-f545-45f4-9bd8-c5ef00630927 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:16.471614 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07868bb0-f545-45f4-9bd8-c5ef00630927 admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:16.471986 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07868bb0-f545-45f4-9bd8-c5ef00630927 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:16.477259 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07868bb0-f545-45f4-9bd8-c5ef00630927 admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:16.478047 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07868bb0-f545-45f4-9bd8-c5ef00630927 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:16.479901 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07868bb0-f545-45f4-9bd8-c5ef00630927 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:16.524874 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62bcda18-d17a-4c1a-aef5-e7bceebebb96 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:16.535987 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 872/1767] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:12:16 2022] GET /identity/v3/roles => generated 1420 bytes in 100 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:12:16.548209 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62bcda18-d17a-4c1a-aef5-e7bceebebb96 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:16.556705 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5a05d2f-bcd3-442d-aaf2-e7c333c71078 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:16.558092 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a05d2f-bcd3-442d-aaf2-e7c333c71078 admin admin] REQUEST_METHOD: `PUT` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:16.558598 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a05d2f-bcd3-442d-aaf2-e7c333c71078 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:16.559053 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a05d2f-bcd3-442d-aaf2-e7c333c71078 admin admin] PATH_INFO: `/v3/projects/428f1bcef936443786e903ba4ea8b87a/users/62c39c1b319643e5b2833086e671b210/roles/6f03d86980c34c36b44a8d6764ff4fda` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:16.592039 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5a05d2f-bcd3-442d-aaf2-e7c333c71078 admin admin] RBAC: Authorizing `identity:create_grant(project_id=428f1bcef936443786e903ba4ea8b87a, user_id=62c39c1b319643e5b2833086e671b210, role_id=6f03d86980c34c36b44a8d6764ff4fda)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:16.595818 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5a05d2f-bcd3-442d-aaf2-e7c333c71078 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:16.619623 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 896/1768] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:12:16 2022] PUT /identity/v3/projects/428f1bcef936443786e903ba4ea8b87a/users/62c39c1b319643e5b2833086e671b210/roles/6f03d86980c34c36b44a8d6764ff4fda => generated 0 bytes in 96 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:12:16.631057 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65a9a1a9-009f-4b9a-bef0-8448ccb99411 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:16.632090 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65a9a1a9-009f-4b9a-bef0-8448ccb99411 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:16.632090 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65a9a1a9-009f-4b9a-bef0-8448ccb99411 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:16.675024 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-65a9a1a9-009f-4b9a-bef0-8448ccb99411 None None] MFA Rules not processed for user `62c39c1b319643e5b2833086e671b210`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:12:16.677054 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-65a9a1a9-009f-4b9a-bef0-8448ccb99411 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:16.787310 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 873/1769] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:12:16 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:12:16.788754 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5a05d2f-bcd3-442d-aaf2-e7c333c71078 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:16.896117 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5a05d2f-bcd3-442d-aaf2-e7c333c71078 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:16.919054 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32551905-5e51-4db1-bd39-d044f198d1eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:16.919897 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32551905-5e51-4db1-bd39-d044f198d1eb admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:16.920324 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32551905-5e51-4db1-bd39-d044f198d1eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:16.920616 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32551905-5e51-4db1-bd39-d044f198d1eb admin admin] PATH_INFO: `/v3/projects` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:16.921484 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32551905-5e51-4db1-bd39-d044f198d1eb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:16.923772 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32551905-5e51-4db1-bd39-d044f198d1eb admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:16.965613 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 897/1770] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:12:16 2022] POST /identity/v3/projects => generated 387 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:12:16.967176 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65a9a1a9-009f-4b9a-bef0-8448ccb99411 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:16.986296 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65a9a1a9-009f-4b9a-bef0-8448ccb99411 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:17.052896 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-955a3d75-1226-4e2b-ae91-2ddb68c96ca4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:17.053508 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-955a3d75-1226-4e2b-ae91-2ddb68c96ca4 admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:17.053787 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-955a3d75-1226-4e2b-ae91-2ddb68c96ca4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:17.054087 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-955a3d75-1226-4e2b-ae91-2ddb68c96ca4 admin admin] PATH_INFO: `/v3/users` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:17.054718 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-955a3d75-1226-4e2b-ae91-2ddb68c96ca4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:17.056672 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-955a3d75-1226-4e2b-ae91-2ddb68c96ca4 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:17.084128 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 874/1771] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:12:16 2022] POST /identity/v3/users => generated 300 bytes in 118 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:12:17.088135 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32551905-5e51-4db1-bd39-d044f198d1eb admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:17.104784 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32551905-5e51-4db1-bd39-d044f198d1eb admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:17.114964 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12b6af7b-fcd0-4fa6-ae5f-dcff08265e1b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:17.115727 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12b6af7b-fcd0-4fa6-ae5f-dcff08265e1b admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:17.117273 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12b6af7b-fcd0-4fa6-ae5f-dcff08265e1b admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:17.117545 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12b6af7b-fcd0-4fa6-ae5f-dcff08265e1b admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:17.123111 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12b6af7b-fcd0-4fa6-ae5f-dcff08265e1b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:17.125495 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12b6af7b-fcd0-4fa6-ae5f-dcff08265e1b admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:17.144712 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 898/1772] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:12:17 2022] GET /identity/v3/roles => generated 1420 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:12:17.155979 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-955a3d75-1226-4e2b-ae91-2ddb68c96ca4 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:17.169435 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-955a3d75-1226-4e2b-ae91-2ddb68c96ca4 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:17.182075 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0858aa0e-8e31-4c88-ad83-0ddde2de9fce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:17.182978 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0858aa0e-8e31-4c88-ad83-0ddde2de9fce admin admin] REQUEST_METHOD: `PUT` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:17.183354 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0858aa0e-8e31-4c88-ad83-0ddde2de9fce admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:17.183742 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0858aa0e-8e31-4c88-ad83-0ddde2de9fce admin admin] PATH_INFO: `/v3/projects/07faa815853248dfbf9b5eb610ac2aa8/users/e86f2c4fe98043989c3d1a7bec3c9416/roles/9d9d5d2c270146488249b7ed943280db` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:17.240954 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0858aa0e-8e31-4c88-ad83-0ddde2de9fce admin admin] RBAC: Authorizing `identity:create_grant(project_id=07faa815853248dfbf9b5eb610ac2aa8, user_id=e86f2c4fe98043989c3d1a7bec3c9416, role_id=9d9d5d2c270146488249b7ed943280db)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:17.244502 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0858aa0e-8e31-4c88-ad83-0ddde2de9fce admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:17.268287 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 875/1773] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:12:17 2022] PUT /identity/v3/projects/07faa815853248dfbf9b5eb610ac2aa8/users/e86f2c4fe98043989c3d1a7bec3c9416/roles/9d9d5d2c270146488249b7ed943280db => generated 0 bytes in 111 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:12:17.271623 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12b6af7b-fcd0-4fa6-ae5f-dcff08265e1b admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:17.282518 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12b6af7b-fcd0-4fa6-ae5f-dcff08265e1b admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:17.408397 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:INSERT=4,keystone:DELETE=3,keystone:SELECT=93 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:17.435335 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a2c0ed9-ff09-4afd-8337-f9da79920dca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:17.436337 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a2c0ed9-ff09-4afd-8337-f9da79920dca admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:17.436822 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a2c0ed9-ff09-4afd-8337-f9da79920dca admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:17.437302 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a2c0ed9-ff09-4afd-8337-f9da79920dca admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:17.438186 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a2c0ed9-ff09-4afd-8337-f9da79920dca admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:17.439917 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a2c0ed9-ff09-4afd-8337-f9da79920dca admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:17.469912 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 899/1774] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:12:17 2022] GET /identity/v3/roles => generated 1420 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:12:17.478175 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0858aa0e-8e31-4c88-ad83-0ddde2de9fce admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:17.492249 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0858aa0e-8e31-4c88-ad83-0ddde2de9fce admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:17.497968 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1c9e31a-cb1e-4f8a-8cfc-5cadeb67fb09 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:17.498883 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c9e31a-cb1e-4f8a-8cfc-5cadeb67fb09 admin admin] REQUEST_METHOD: `PUT` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:17.499270 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c9e31a-cb1e-4f8a-8cfc-5cadeb67fb09 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:17.499945 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c9e31a-cb1e-4f8a-8cfc-5cadeb67fb09 admin admin] PATH_INFO: `/v3/projects/07faa815853248dfbf9b5eb610ac2aa8/users/e86f2c4fe98043989c3d1a7bec3c9416/roles/9d9d5d2c270146488249b7ed943280db` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:17.505270 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1c9e31a-cb1e-4f8a-8cfc-5cadeb67fb09 admin admin] RBAC: Authorizing `identity:create_grant(project_id=07faa815853248dfbf9b5eb610ac2aa8, user_id=e86f2c4fe98043989c3d1a7bec3c9416, role_id=9d9d5d2c270146488249b7ed943280db)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:17.508559 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1c9e31a-cb1e-4f8a-8cfc-5cadeb67fb09 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:17.544355 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 876/1775] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:12:17 2022] PUT /identity/v3/projects/07faa815853248dfbf9b5eb610ac2aa8/users/e86f2c4fe98043989c3d1a7bec3c9416/roles/9d9d5d2c270146488249b7ed943280db => generated 0 bytes in 69 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:12:17.546443 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33e3b159-8194-4d36-b9ac-cb4e365110af None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:17.546981 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33e3b159-8194-4d36-b9ac-cb4e365110af None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:17.547316 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33e3b159-8194-4d36-b9ac-cb4e365110af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:17.585611 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-33e3b159-8194-4d36-b9ac-cb4e365110af None None] MFA Rules not processed for user `e86f2c4fe98043989c3d1a7bec3c9416`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:12:17.587423 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-33e3b159-8194-4d36-b9ac-cb4e365110af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:17.712668 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 900/1776] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:12:17 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:12:17.732310 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1c9e31a-cb1e-4f8a-8cfc-5cadeb67fb09 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:17.738428 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-d1c9e31a-cb1e-4f8a-8cfc-5cadeb67fb09 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:17.766392 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33e3b159-8194-4d36-b9ac-cb4e365110af None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:17.768259 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-33e3b159-8194-4d36-b9ac-cb4e365110af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:17.822246 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:17.862936 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1c9e31a-cb1e-4f8a-8cfc-5cadeb67fb09 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:17.866733 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b052817b-33ba-4584-bbbc-d36e6cdadeef service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '019a79b2affe478b8a30d1c538d0415c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:17.867554 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b052817b-33ba-4584-bbbc-d36e6cdadeef service cinder] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:17.868007 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b052817b-33ba-4584-bbbc-d36e6cdadeef service cinder] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:17.868425 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b052817b-33ba-4584-bbbc-d36e6cdadeef service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:17.879825 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b052817b-33ba-4584-bbbc-d36e6cdadeef service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:17.880502 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33e3b159-8194-4d36-b9ac-cb4e365110af None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:17.884215 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b052817b-33ba-4584-bbbc-d36e6cdadeef service cinder] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:17.900517 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86ec68ce-96c3-42ef-832c-72ce8a90fcbf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:17.902923 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86ec68ce-96c3-42ef-832c-72ce8a90fcbf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:17.903210 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86ec68ce-96c3-42ef-832c-72ce8a90fcbf admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:17.903490 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86ec68ce-96c3-42ef-832c-72ce8a90fcbf admin admin] PATH_INFO: `/v3/users/5e0615e5403b4e47937426a43291d704` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:17.905255 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86ec68ce-96c3-42ef-832c-72ce8a90fcbf admin admin] RBAC: Authorizing `identity:delete_user(user_id=5e0615e5403b4e47937426a43291d704)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:17.907339 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86ec68ce-96c3-42ef-832c-72ce8a90fcbf admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:17.913690 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 877/1777] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:12:17 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 184 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:12:17.948486 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b052817b-33ba-4584-bbbc-d36e6cdadeef service cinder] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:17.975311 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b052817b-33ba-4584-bbbc-d36e6cdadeef service cinder] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:17.980533 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87d022dc-6769-4351-a733-7d15c5fa5d3e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:17.981161 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d022dc-6769-4351-a733-7d15c5fa5d3e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:17.981441 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d022dc-6769-4351-a733-7d15c5fa5d3e admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:17.981798 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d022dc-6769-4351-a733-7d15c5fa5d3e admin admin] PATH_INFO: `/v3/users/62c39c1b319643e5b2833086e671b210` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:17.983744 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87d022dc-6769-4351-a733-7d15c5fa5d3e admin admin] RBAC: Authorizing `identity:delete_user(user_id=62c39c1b319643e5b2833086e671b210)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:17.986039 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87d022dc-6769-4351-a733-7d15c5fa5d3e admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:18.011975 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-86ec68ce-96c3-42ef-832c-72ce8a90fcbf admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5e0615e5403b4e47937426a43291d704'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:18.022664 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-86ec68ce-96c3-42ef-832c-72ce8a90fcbf admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5e0615e5403b4e47937426a43291d704'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:18.048834 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-86ec68ce-96c3-42ef-832c-72ce8a90fcbf admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5e0615e5403b4e47937426a43291d704'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:18.051595 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:18.060460 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-86ec68ce-96c3-42ef-832c-72ce8a90fcbf admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5e0615e5403b4e47937426a43291d704'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:18.063208 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-86ec68ce-96c3-42ef-832c-72ce8a90fcbf admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5e0615e5403b4e47937426a43291d704'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:18.076715 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 901/1778] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:12:17 2022] DELETE /identity/v3/users/5e0615e5403b4e47937426a43291d704 => generated 0 bytes in 307 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 05:12:18.078751 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e34ddff-fbad-49c3-9996-318eec29feaf None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:18.079241 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e34ddff-fbad-49c3-9996-318eec29feaf None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:18.079691 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e34ddff-fbad-49c3-9996-318eec29feaf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:18.106529 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-2e34ddff-fbad-49c3-9996-318eec29feaf None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:12:18.108014 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-87d022dc-6769-4351-a733-7d15c5fa5d3e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '62c39c1b319643e5b2833086e671b210'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:18.109041 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-87d022dc-6769-4351-a733-7d15c5fa5d3e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '62c39c1b319643e5b2833086e671b210'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:18.110317 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-2e34ddff-fbad-49c3-9996-318eec29feaf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:18.110804 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-87d022dc-6769-4351-a733-7d15c5fa5d3e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '62c39c1b319643e5b2833086e671b210'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:18.136213 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-87d022dc-6769-4351-a733-7d15c5fa5d3e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '62c39c1b319643e5b2833086e671b210'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:18.158490 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-87d022dc-6769-4351-a733-7d15c5fa5d3e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '62c39c1b319643e5b2833086e671b210'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:18.175000 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 878/1779] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:12:17 2022] DELETE /identity/v3/users/62c39c1b319643e5b2833086e671b210 => generated 0 bytes in 227 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 05:12:18.179757 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e2106ba-8023-48bf-b399-74f962414303 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:18.180155 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e2106ba-8023-48bf-b399-74f962414303 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:18.180437 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e2106ba-8023-48bf-b399-74f962414303 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:18.201679 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-5e2106ba-8023-48bf-b399-74f962414303 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:12:18.203066 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-5e2106ba-8023-48bf-b399-74f962414303 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:18.222905 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 902/1780] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:12:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:12:18.230675 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e34ddff-fbad-49c3-9996-318eec29feaf None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:18.233123 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-2e34ddff-fbad-49c3-9996-318eec29feaf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:18.287255 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 879/1781] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:12:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 05:12:18.297142 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e2106ba-8023-48bf-b399-74f962414303 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:18.298621 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-5e2106ba-8023-48bf-b399-74f962414303 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:18.304824 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e34ddff-fbad-49c3-9996-318eec29feaf None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:18.363912 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:18.368418 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e2106ba-8023-48bf-b399-74f962414303 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:18.373638 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3636319a-b582-4524-ba1f-0c84f79d91dd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:18.374617 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3636319a-b582-4524-ba1f-0c84f79d91dd service neutron] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:18.375470 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3636319a-b582-4524-ba1f-0c84f79d91dd service neutron] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:18.376121 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3636319a-b582-4524-ba1f-0c84f79d91dd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:18.380129 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd2696fd-c2f1-46a0-8a41-3b79a7444572 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:18.380747 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd2696fd-c2f1-46a0-8a41-3b79a7444572 service neutron] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:18.381072 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd2696fd-c2f1-46a0-8a41-3b79a7444572 service neutron] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:18.381353 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd2696fd-c2f1-46a0-8a41-3b79a7444572 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:18.388564 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3636319a-b582-4524-ba1f-0c84f79d91dd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:18.389890 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3636319a-b582-4524-ba1f-0c84f79d91dd service neutron] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:18.396959 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd2696fd-c2f1-46a0-8a41-3b79a7444572 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:18.398482 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd2696fd-c2f1-46a0-8a41-3b79a7444572 service neutron] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:18.412376 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 903/1782] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:12:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 182 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:12:18.427868 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 880/1783] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:12:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 129 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 05:12:19.032812 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd2696fd-c2f1-46a0-8a41-3b79a7444572 service neutron] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:19.035661 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-dd2696fd-c2f1-46a0-8a41-3b79a7444572 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:19.043974 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3636319a-b582-4524-ba1f-0c84f79d91dd service neutron] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:19.046886 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-3636319a-b582-4524-ba1f-0c84f79d91dd service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:19.069286 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3636319a-b582-4524-ba1f-0c84f79d91dd service neutron] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:19.083993 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd2696fd-c2f1-46a0-8a41-3b79a7444572 service neutron] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:19.085269 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecd67591-2acf-46f1-b4f7-5ff4403ad7b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:19.086062 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecd67591-2acf-46f1-b4f7-5ff4403ad7b3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:19.086420 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecd67591-2acf-46f1-b4f7-5ff4403ad7b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:19.086892 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecd67591-2acf-46f1-b4f7-5ff4403ad7b3 admin admin] PATH_INFO: `/v3/projects/428f1bcef936443786e903ba4ea8b87a` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:19.088283 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecd67591-2acf-46f1-b4f7-5ff4403ad7b3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=428f1bcef936443786e903ba4ea8b87a)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:19.090409 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecd67591-2acf-46f1-b4f7-5ff4403ad7b3 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:19.100312 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0560f23-7edd-4fca-aa17-797ced09d6aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:19.101681 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0560f23-7edd-4fca-aa17-797ced09d6aa admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:19.103432 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0560f23-7edd-4fca-aa17-797ced09d6aa admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:19.109638 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0560f23-7edd-4fca-aa17-797ced09d6aa admin admin] PATH_INFO: `/v3/projects/7534ac52c2524faaabb38edb8d476695` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:19.117275 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0560f23-7edd-4fca-aa17-797ced09d6aa admin admin] RBAC: Authorizing `identity:delete_project(project_id=7534ac52c2524faaabb38edb8d476695)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:19.135001 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0560f23-7edd-4fca-aa17-797ced09d6aa admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:19.208567 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:19.213433 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-a0560f23-7edd-4fca-aa17-797ced09d6aa admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7534ac52c2524faaabb38edb8d476695', 'request_id': 'req-a0560f23-7edd-4fca-aa17-797ced09d6aa'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:19.224412 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-a0560f23-7edd-4fca-aa17-797ced09d6aa admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7534ac52c2524faaabb38edb8d476695', 'request_id': 'req-a0560f23-7edd-4fca-aa17-797ced09d6aa'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:19.232302 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-ecd67591-2acf-46f1-b4f7-5ff4403ad7b3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '428f1bcef936443786e903ba4ea8b87a', 'request_id': 'req-ecd67591-2acf-46f1-b4f7-5ff4403ad7b3'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:19.233641 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-a0560f23-7edd-4fca-aa17-797ced09d6aa admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7534ac52c2524faaabb38edb8d476695', 'request_id': 'req-a0560f23-7edd-4fca-aa17-797ced09d6aa'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:19.252143 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-ecd67591-2acf-46f1-b4f7-5ff4403ad7b3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '428f1bcef936443786e903ba4ea8b87a', 'request_id': 'req-ecd67591-2acf-46f1-b4f7-5ff4403ad7b3'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:19.254803 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-a0560f23-7edd-4fca-aa17-797ced09d6aa admin admin] The token cache is being invalidate because project 7534ac52c2524faaabb38edb8d476695 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81625) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:12:19.256133 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-a0560f23-7edd-4fca-aa17-797ced09d6aa admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:19.260837 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-a0560f23-7edd-4fca-aa17-797ced09d6aa admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:19.274166 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-ecd67591-2acf-46f1-b4f7-5ff4403ad7b3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '428f1bcef936443786e903ba4ea8b87a', 'request_id': 'req-ecd67591-2acf-46f1-b4f7-5ff4403ad7b3'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:19.277574 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 904/1784] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:12:19 2022] DELETE /identity/v3/projects/7534ac52c2524faaabb38edb8d476695 => generated 0 bytes in 245 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:12:19.281435 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0560f23-7edd-4fca-aa17-797ced09d6aa admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:19.286114 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-a0560f23-7edd-4fca-aa17-797ced09d6aa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:19.308100 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-ecd67591-2acf-46f1-b4f7-5ff4403ad7b3 admin admin] The token cache is being invalidate because project 428f1bcef936443786e903ba4ea8b87a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81624) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:12:19.308527 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-ecd67591-2acf-46f1-b4f7-5ff4403ad7b3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:19.310531 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-ecd67591-2acf-46f1-b4f7-5ff4403ad7b3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:19.314375 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 881/1785] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:12:19 2022] DELETE /identity/v3/projects/428f1bcef936443786e903ba4ea8b87a => generated 0 bytes in 270 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 05:12:19.317844 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecd67591-2acf-46f1-b4f7-5ff4403ad7b3 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:19.322865 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-ecd67591-2acf-46f1-b4f7-5ff4403ad7b3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:19.405971 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0560f23-7edd-4fca-aa17-797ced09d6aa admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:19.421051 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16c94cb9-ddf3-45fe-ab70-934592573056 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:19.422033 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c94cb9-ddf3-45fe-ab70-934592573056 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:19.422449 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c94cb9-ddf3-45fe-ab70-934592573056 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:19.422873 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c94cb9-ddf3-45fe-ab70-934592573056 admin admin] PATH_INFO: `/v3/users/80188bcd72894a7797d995c6cfdf54e3` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:19.424912 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16c94cb9-ddf3-45fe-ab70-934592573056 admin admin] RBAC: Authorizing `identity:delete_user(user_id=80188bcd72894a7797d995c6cfdf54e3)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:19.428153 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16c94cb9-ddf3-45fe-ab70-934592573056 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:19.489705 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecd67591-2acf-46f1-b4f7-5ff4403ad7b3 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:19.496257 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e757700-e5a3-4975-af3e-79f21fafbb19 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:19.496939 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e757700-e5a3-4975-af3e-79f21fafbb19 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:19.497447 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e757700-e5a3-4975-af3e-79f21fafbb19 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:19.497727 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e757700-e5a3-4975-af3e-79f21fafbb19 admin admin] PATH_INFO: `/v3/users/e86f2c4fe98043989c3d1a7bec3c9416` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:19.499069 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e757700-e5a3-4975-af3e-79f21fafbb19 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e86f2c4fe98043989c3d1a7bec3c9416)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:19.501363 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e757700-e5a3-4975-af3e-79f21fafbb19 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:19.563376 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-16c94cb9-ddf3-45fe-ab70-934592573056 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '80188bcd72894a7797d995c6cfdf54e3'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:19.572664 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-1e757700-e5a3-4975-af3e-79f21fafbb19 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e86f2c4fe98043989c3d1a7bec3c9416'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:19.573133 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-16c94cb9-ddf3-45fe-ab70-934592573056 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '80188bcd72894a7797d995c6cfdf54e3'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:19.573543 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-1e757700-e5a3-4975-af3e-79f21fafbb19 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e86f2c4fe98043989c3d1a7bec3c9416'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:19.574337 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-1e757700-e5a3-4975-af3e-79f21fafbb19 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e86f2c4fe98043989c3d1a7bec3c9416'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:19.593785 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-1e757700-e5a3-4975-af3e-79f21fafbb19 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e86f2c4fe98043989c3d1a7bec3c9416'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:19.594379 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-16c94cb9-ddf3-45fe-ab70-934592573056 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '80188bcd72894a7797d995c6cfdf54e3'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:19.604748 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-1e757700-e5a3-4975-af3e-79f21fafbb19 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e86f2c4fe98043989c3d1a7bec3c9416'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:19.606855 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-16c94cb9-ddf3-45fe-ab70-934592573056 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '80188bcd72894a7797d995c6cfdf54e3'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:19.607612 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-16c94cb9-ddf3-45fe-ab70-934592573056 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '80188bcd72894a7797d995c6cfdf54e3'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:19.608447 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=82,keystone:DELETE=16,keystone:INSERT=2 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:19.615698 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 905/1786] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:12:19 2022] DELETE /identity/v3/users/80188bcd72894a7797d995c6cfdf54e3 => generated 0 bytes in 335 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 05:12:19.624215 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 882/1787] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:12:19 2022] DELETE /identity/v3/users/e86f2c4fe98043989c3d1a7bec3c9416 => generated 0 bytes in 307 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:12:19.990670 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16c94cb9-ddf3-45fe-ab70-934592573056 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:19.992425 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-16c94cb9-ddf3-45fe-ab70-934592573056 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:20.015420 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e757700-e5a3-4975-af3e-79f21fafbb19 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:20.016925 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-1e757700-e5a3-4975-af3e-79f21fafbb19 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:20.122259 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:20.132663 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e757700-e5a3-4975-af3e-79f21fafbb19 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:20.140662 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbc880de-8bb2-4c9a-98d6-94b96160914b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:20.141386 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbc880de-8bb2-4c9a-98d6-94b96160914b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:20.141742 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbc880de-8bb2-4c9a-98d6-94b96160914b admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:20.141887 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16c94cb9-ddf3-45fe-ab70-934592573056 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:20.142165 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbc880de-8bb2-4c9a-98d6-94b96160914b admin admin] PATH_INFO: `/v3/projects/07faa815853248dfbf9b5eb610ac2aa8` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:20.143434 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbc880de-8bb2-4c9a-98d6-94b96160914b admin admin] RBAC: Authorizing `identity:delete_project(project_id=07faa815853248dfbf9b5eb610ac2aa8)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:20.146433 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbc880de-8bb2-4c9a-98d6-94b96160914b admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:20.149675 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e44717a-6ba5-4134-ac12-3c33405dce76 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:20.150313 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e44717a-6ba5-4134-ac12-3c33405dce76 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:20.150592 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e44717a-6ba5-4134-ac12-3c33405dce76 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:20.150888 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e44717a-6ba5-4134-ac12-3c33405dce76 admin admin] PATH_INFO: `/v3/projects/d4e7dd205fc948898d3a0004183a0f06` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:20.152064 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e44717a-6ba5-4134-ac12-3c33405dce76 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d4e7dd205fc948898d3a0004183a0f06)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:20.154707 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e44717a-6ba5-4134-ac12-3c33405dce76 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:20.213894 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-dbc880de-8bb2-4c9a-98d6-94b96160914b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '07faa815853248dfbf9b5eb610ac2aa8', 'request_id': 'req-dbc880de-8bb2-4c9a-98d6-94b96160914b'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:20.225013 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-dbc880de-8bb2-4c9a-98d6-94b96160914b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '07faa815853248dfbf9b5eb610ac2aa8', 'request_id': 'req-dbc880de-8bb2-4c9a-98d6-94b96160914b'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:20.231546 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-7e44717a-6ba5-4134-ac12-3c33405dce76 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd4e7dd205fc948898d3a0004183a0f06', 'request_id': 'req-7e44717a-6ba5-4134-ac12-3c33405dce76'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:20.234395 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-dbc880de-8bb2-4c9a-98d6-94b96160914b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '07faa815853248dfbf9b5eb610ac2aa8', 'request_id': 'req-dbc880de-8bb2-4c9a-98d6-94b96160914b'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:20.241390 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-7e44717a-6ba5-4134-ac12-3c33405dce76 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd4e7dd205fc948898d3a0004183a0f06', 'request_id': 'req-7e44717a-6ba5-4134-ac12-3c33405dce76'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:20.251506 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-7e44717a-6ba5-4134-ac12-3c33405dce76 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd4e7dd205fc948898d3a0004183a0f06', 'request_id': 'req-7e44717a-6ba5-4134-ac12-3c33405dce76'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:20.256975 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-dbc880de-8bb2-4c9a-98d6-94b96160914b admin admin] The token cache is being invalidate because project 07faa815853248dfbf9b5eb610ac2aa8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81624) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:12:20.257287 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-dbc880de-8bb2-4c9a-98d6-94b96160914b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:20.263246 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-dbc880de-8bb2-4c9a-98d6-94b96160914b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:20.275236 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-7e44717a-6ba5-4134-ac12-3c33405dce76 admin admin] The token cache is being invalidate because project d4e7dd205fc948898d3a0004183a0f06 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81625) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:12:20.275822 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-7e44717a-6ba5-4134-ac12-3c33405dce76 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:20.278181 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 883/1788] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:12:20 2022] DELETE /identity/v3/projects/07faa815853248dfbf9b5eb610ac2aa8 => generated 0 bytes in 260 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:12:20.278601 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-7e44717a-6ba5-4134-ac12-3c33405dce76 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:20.280506 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b05e653a-e9df-4cbb-8023-68d372b24ba6 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:20.280801 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b05e653a-e9df-4cbb-8023-68d372b24ba6 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:20.281073 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b05e653a-e9df-4cbb-8023-68d372b24ba6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:20.285445 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 906/1789] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:12:19 2022] DELETE /identity/v3/projects/d4e7dd205fc948898d3a0004183a0f06 => generated 0 bytes in 295 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:12:20.292482 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de82a9b6-03a9-41db-8095-fac7b0a719d6 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:20.293016 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de82a9b6-03a9-41db-8095-fac7b0a719d6 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:20.293482 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de82a9b6-03a9-41db-8095-fac7b0a719d6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:20.317278 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-b05e653a-e9df-4cbb-8023-68d372b24ba6 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:12:20.318513 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-b05e653a-e9df-4cbb-8023-68d372b24ba6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:20.321497 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-de82a9b6-03a9-41db-8095-fac7b0a719d6 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:12:20.322958 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-de82a9b6-03a9-41db-8095-fac7b0a719d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:20.404796 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 907/1790] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:12:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:12:20.406343 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 884/1791] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:12:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:12:20.413358 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80acc14d-c629-4bb6-9003-f8435d7c9cbb None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:20.413358 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80acc14d-c629-4bb6-9003-f8435d7c9cbb None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:20.413358 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80acc14d-c629-4bb6-9003-f8435d7c9cbb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:20.415025 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba09cc4f-93b2-44b2-9540-4f7aba31a047 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:20.415371 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba09cc4f-93b2-44b2-9540-4f7aba31a047 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:20.416050 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba09cc4f-93b2-44b2-9540-4f7aba31a047 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:20.457682 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-ba09cc4f-93b2-44b2-9540-4f7aba31a047 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:12:20.464293 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-ba09cc4f-93b2-44b2-9540-4f7aba31a047 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:20.475037 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-80acc14d-c629-4bb6-9003-f8435d7c9cbb None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:12:20.476302 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-80acc14d-c629-4bb6-9003-f8435d7c9cbb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:20.478443 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 885/1792] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:12:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 64 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:12:20.485745 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba09cc4f-93b2-44b2-9540-4f7aba31a047 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:20.497074 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 908/1793] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:12:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 05:12:20.504419 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80acc14d-c629-4bb6-9003-f8435d7c9cbb None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:20.505070 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba09cc4f-93b2-44b2-9540-4f7aba31a047 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:20.514435 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2029e893-9c45-41d2-8701-3f14dee908f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:20.515414 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2029e893-9c45-41d2-8701-3f14dee908f7 admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:20.515891 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2029e893-9c45-41d2-8701-3f14dee908f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:20.516366 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2029e893-9c45-41d2-8701-3f14dee908f7 admin admin] PATH_INFO: `/v3/domains` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:20.517179 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80acc14d-c629-4bb6-9003-f8435d7c9cbb None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:20.517403 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2029e893-9c45-41d2-8701-3f14dee908f7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:12:20.517997 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2029e893-9c45-41d2-8701-3f14dee908f7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:20.520185 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2029e893-9c45-41d2-8701-3f14dee908f7 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:20.529305 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61e629c4-3b7d-482a-96f4-7a71e2886baf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:20.529305 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e629c4-3b7d-482a-96f4-7a71e2886baf admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:20.529305 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e629c4-3b7d-482a-96f4-7a71e2886baf admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:20.529305 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e629c4-3b7d-482a-96f4-7a71e2886baf admin admin] PATH_INFO: `/v3/domains` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:20.530058 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61e629c4-3b7d-482a-96f4-7a71e2886baf admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:12:20.530058 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61e629c4-3b7d-482a-96f4-7a71e2886baf admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:20.546863 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 886/1794] 10.222.0.77 () {58 vars in 1153 bytes} [Tue Aug 30 05:12:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 61 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:12:20.553106 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61e629c4-3b7d-482a-96f4-7a71e2886baf admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:20.555319 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2029e893-9c45-41d2-8701-3f14dee908f7 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:20.562479 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:DELETE=8,keystone:SELECT=91,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:20.569061 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2029e893-9c45-41d2-8701-3f14dee908f7 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:20.570774 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 909/1795] 10.222.0.77 () {58 vars in 1153 bytes} [Tue Aug 30 05:12:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 67 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:12:20.571836 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61e629c4-3b7d-482a-96f4-7a71e2886baf admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:20.581123 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-354c48cd-8f28-493c-b0aa-0c46e9f32df6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:20.582373 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-354c48cd-8f28-493c-b0aa-0c46e9f32df6 admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:20.583090 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-354c48cd-8f28-493c-b0aa-0c46e9f32df6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:20.583645 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-354c48cd-8f28-493c-b0aa-0c46e9f32df6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:20.584791 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-354c48cd-8f28-493c-b0aa-0c46e9f32df6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:20.587817 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-354c48cd-8f28-493c-b0aa-0c46e9f32df6 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:20.588184 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61e629c4-3b7d-482a-96f4-7a71e2886baf admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:20.593076 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4761587-09c0-414d-8785-51d2afde2e29 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:20.593725 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4761587-09c0-414d-8785-51d2afde2e29 admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:20.594008 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4761587-09c0-414d-8785-51d2afde2e29 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:20.594366 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4761587-09c0-414d-8785-51d2afde2e29 admin admin] PATH_INFO: `/v3/projects` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:20.595423 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4761587-09c0-414d-8785-51d2afde2e29 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:20.597806 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4761587-09c0-414d-8785-51d2afde2e29 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:20.629626 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 887/1796] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:12:20 2022] POST /identity/v3/projects => generated 385 bytes in 79 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 05:12:20.633616 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-354c48cd-8f28-493c-b0aa-0c46e9f32df6 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:20.642864 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 910/1797] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:12:20 2022] POST /identity/v3/projects => generated 385 bytes in 63 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:12:20.642864 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4761587-09c0-414d-8785-51d2afde2e29 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:20.686781 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:20.731781 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4761587-09c0-414d-8785-51d2afde2e29 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:20.737362 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1efa0f26-dd40-4d28-a052-c15d9567a6eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:20.738856 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1efa0f26-dd40-4d28-a052-c15d9567a6eb admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:20.739016 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-354c48cd-8f28-493c-b0aa-0c46e9f32df6 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:20.739271 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1efa0f26-dd40-4d28-a052-c15d9567a6eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:20.739502 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1efa0f26-dd40-4d28-a052-c15d9567a6eb admin admin] PATH_INFO: `/v3/users` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:20.740183 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1efa0f26-dd40-4d28-a052-c15d9567a6eb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:20.742163 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1efa0f26-dd40-4d28-a052-c15d9567a6eb admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:20.744155 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df7380af-8b52-40c0-a6df-dbe41a945754 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:20.744770 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df7380af-8b52-40c0-a6df-dbe41a945754 admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:20.745048 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df7380af-8b52-40c0-a6df-dbe41a945754 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:20.745319 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df7380af-8b52-40c0-a6df-dbe41a945754 admin admin] PATH_INFO: `/v3/users` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:20.745947 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df7380af-8b52-40c0-a6df-dbe41a945754 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:20.748091 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df7380af-8b52-40c0-a6df-dbe41a945754 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:20.771175 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 911/1798] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:12:20 2022] POST /identity/v3/users => generated 293 bytes in 136 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:12:20.774966 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1efa0f26-dd40-4d28-a052-c15d9567a6eb admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:20.775726 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 888/1799] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:12:20 2022] POST /identity/v3/users => generated 293 bytes in 143 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 05:12:20.779573 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df7380af-8b52-40c0-a6df-dbe41a945754 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:20.788220 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1efa0f26-dd40-4d28-a052-c15d9567a6eb admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:20.789958 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df7380af-8b52-40c0-a6df-dbe41a945754 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:20.793373 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0145ee89-b902-4ce5-8bf6-a1e0e136ee26 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:20.794013 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0145ee89-b902-4ce5-8bf6-a1e0e136ee26 admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:20.794347 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0145ee89-b902-4ce5-8bf6-a1e0e136ee26 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:20.794623 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0145ee89-b902-4ce5-8bf6-a1e0e136ee26 admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:20.794916 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-696a71aa-1b42-4221-a3dc-83780fb20434 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:20.795307 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0145ee89-b902-4ce5-8bf6-a1e0e136ee26 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:20.795663 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-696a71aa-1b42-4221-a3dc-83780fb20434 admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:20.796013 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-696a71aa-1b42-4221-a3dc-83780fb20434 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:20.796353 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-696a71aa-1b42-4221-a3dc-83780fb20434 admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:20.796911 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0145ee89-b902-4ce5-8bf6-a1e0e136ee26 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:20.797062 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-696a71aa-1b42-4221-a3dc-83780fb20434 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:20.798783 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-696a71aa-1b42-4221-a3dc-83780fb20434 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:20.810089 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 889/1800] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:12:20 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 05:12:20.812677 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 912/1801] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:12:20 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 05:12:20.814031 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-696a71aa-1b42-4221-a3dc-83780fb20434 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:20.816592 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0145ee89-b902-4ce5-8bf6-a1e0e136ee26 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:20.827588 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-696a71aa-1b42-4221-a3dc-83780fb20434 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:20.833804 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0145ee89-b902-4ce5-8bf6-a1e0e136ee26 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:20.839422 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47a484ee-ec9b-47cb-85f3-745319ad9832 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:20.840004 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f49f07e-4fc0-4df3-9be7-02bc8a2f112f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:20.840179 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47a484ee-ec9b-47cb-85f3-745319ad9832 admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:20.840550 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47a484ee-ec9b-47cb-85f3-745319ad9832 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:20.840645 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f49f07e-4fc0-4df3-9be7-02bc8a2f112f admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:20.840907 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f49f07e-4fc0-4df3-9be7-02bc8a2f112f admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:20.841003 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47a484ee-ec9b-47cb-85f3-745319ad9832 admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:20.841626 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47a484ee-ec9b-47cb-85f3-745319ad9832 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:20.842871 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f49f07e-4fc0-4df3-9be7-02bc8a2f112f admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:20.843416 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47a484ee-ec9b-47cb-85f3-745319ad9832 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:20.844241 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f49f07e-4fc0-4df3-9be7-02bc8a2f112f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:20.846128 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f49f07e-4fc0-4df3-9be7-02bc8a2f112f admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:20.863360 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 913/1802] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:12:20 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:12:20.869835 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 890/1803] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:12:20 2022] GET /identity/v3/roles => generated 1420 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:12:20.870989 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47a484ee-ec9b-47cb-85f3-745319ad9832 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:20.874387 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f49f07e-4fc0-4df3-9be7-02bc8a2f112f admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:20.882645 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47a484ee-ec9b-47cb-85f3-745319ad9832 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:20.889127 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40cc44c1-8d7b-4427-ae9f-167937b48a39 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:20.889811 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40cc44c1-8d7b-4427-ae9f-167937b48a39 admin admin] REQUEST_METHOD: `PUT` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:20.890158 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40cc44c1-8d7b-4427-ae9f-167937b48a39 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:20.890680 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40cc44c1-8d7b-4427-ae9f-167937b48a39 admin admin] PATH_INFO: `/v3/projects/a7c46125ea924a3aaebdd47f16ef5de6/users/b174111182d04974a14f44c96b178dd4/roles/6f03d86980c34c36b44a8d6764ff4fda` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:20.895199 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f49f07e-4fc0-4df3-9be7-02bc8a2f112f admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:20.901485 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c750e52-53cd-4ede-b2c2-7583781c7375 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:20.902648 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c750e52-53cd-4ede-b2c2-7583781c7375 admin admin] REQUEST_METHOD: `PUT` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:20.903175 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c750e52-53cd-4ede-b2c2-7583781c7375 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:20.903687 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c750e52-53cd-4ede-b2c2-7583781c7375 admin admin] PATH_INFO: `/v3/projects/0f41c6bcddee420791540051817ac6d5/users/2cecbe6f7aa043a79ce11446caaea33b/roles/6f03d86980c34c36b44a8d6764ff4fda` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:20.909221 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40cc44c1-8d7b-4427-ae9f-167937b48a39 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a7c46125ea924a3aaebdd47f16ef5de6, user_id=b174111182d04974a14f44c96b178dd4, role_id=6f03d86980c34c36b44a8d6764ff4fda)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:20.912885 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40cc44c1-8d7b-4427-ae9f-167937b48a39 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:20.934587 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 914/1804] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:12:20 2022] PUT /identity/v3/projects/a7c46125ea924a3aaebdd47f16ef5de6/users/b174111182d04974a14f44c96b178dd4/roles/6f03d86980c34c36b44a8d6764ff4fda => generated 0 bytes in 67 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 05:12:20.935609 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c750e52-53cd-4ede-b2c2-7583781c7375 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0f41c6bcddee420791540051817ac6d5, user_id=2cecbe6f7aa043a79ce11446caaea33b, role_id=6f03d86980c34c36b44a8d6764ff4fda)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:20.938798 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c750e52-53cd-4ede-b2c2-7583781c7375 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:20.945679 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04732382-c4f0-46a5-9276-ed318fdcc5c0 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:20.946011 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04732382-c4f0-46a5-9276-ed318fdcc5c0 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:20.946285 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04732382-c4f0-46a5-9276-ed318fdcc5c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:20.964917 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 891/1805] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:12:20 2022] PUT /identity/v3/projects/0f41c6bcddee420791540051817ac6d5/users/2cecbe6f7aa043a79ce11446caaea33b/roles/6f03d86980c34c36b44a8d6764ff4fda => generated 0 bytes in 91 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 05:12:20.979200 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c28ab2-54e2-4b14-9c78-6331efc3949b None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:20.979729 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c28ab2-54e2-4b14-9c78-6331efc3949b None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:20.980150 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c28ab2-54e2-4b14-9c78-6331efc3949b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:20.990614 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-04732382-c4f0-46a5-9276-ed318fdcc5c0 None None] MFA Rules not processed for user `b174111182d04974a14f44c96b178dd4`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:12:20.991509 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-04732382-c4f0-46a5-9276-ed318fdcc5c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:21.014098 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-02c28ab2-54e2-4b14-9c78-6331efc3949b None None] MFA Rules not processed for user `2cecbe6f7aa043a79ce11446caaea33b`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:12:21.015853 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-02c28ab2-54e2-4b14-9c78-6331efc3949b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:21.077133 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 915/1806] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:12:20 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 05:12:21.083864 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04732382-c4f0-46a5-9276-ed318fdcc5c0 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:21.146074 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 892/1807] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:12:20 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:12:21.151764 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02c28ab2-54e2-4b14-9c78-6331efc3949b None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:21.173150 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:21.174750 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04732382-c4f0-46a5-9276-ed318fdcc5c0 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:21.197869 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-879b302b-a53d-4573-a43e-7215a6c5e7d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:21.198494 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-879b302b-a53d-4573-a43e-7215a6c5e7d5 admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:21.198773 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-879b302b-a53d-4573-a43e-7215a6c5e7d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:21.199073 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-879b302b-a53d-4573-a43e-7215a6c5e7d5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:21.199710 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-879b302b-a53d-4573-a43e-7215a6c5e7d5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:21.201627 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-879b302b-a53d-4573-a43e-7215a6c5e7d5 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:21.236109 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:21.245054 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02c28ab2-54e2-4b14-9c78-6331efc3949b None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:21.250158 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c231170-dd52-4270-ae4f-bcaae3e87676 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:21.250785 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c231170-dd52-4270-ae4f-bcaae3e87676 admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:21.251110 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c231170-dd52-4270-ae4f-bcaae3e87676 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:21.251385 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c231170-dd52-4270-ae4f-bcaae3e87676 admin admin] PATH_INFO: `/v3/projects` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:21.252028 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c231170-dd52-4270-ae4f-bcaae3e87676 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:21.253046 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 916/1808] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:12:21 2022] POST /identity/v3/projects => generated 383 bytes in 170 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:12:21.254477 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c231170-dd52-4270-ae4f-bcaae3e87676 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:21.257262 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-879b302b-a53d-4573-a43e-7215a6c5e7d5 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:21.324166 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 893/1809] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:12:21 2022] POST /identity/v3/projects => generated 387 bytes in 174 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 05:12:21.325242 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c231170-dd52-4270-ae4f-bcaae3e87676 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:21.416152 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-879b302b-a53d-4573-a43e-7215a6c5e7d5 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:21.478369 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c231170-dd52-4270-ae4f-bcaae3e87676 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:21.495874 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35d42b53-568b-4507-9710-e5099eac3a1c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:21.496493 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d42b53-568b-4507-9710-e5099eac3a1c admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:21.496769 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d42b53-568b-4507-9710-e5099eac3a1c admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:21.497039 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d42b53-568b-4507-9710-e5099eac3a1c admin admin] PATH_INFO: `/v3/users` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:21.497677 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35d42b53-568b-4507-9710-e5099eac3a1c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:21.499636 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35d42b53-568b-4507-9710-e5099eac3a1c admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:21.522291 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab8818a1-ffae-40f2-ae0c-5efed46336c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:21.523581 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab8818a1-ffae-40f2-ae0c-5efed46336c3 admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:21.523921 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab8818a1-ffae-40f2-ae0c-5efed46336c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:21.524192 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab8818a1-ffae-40f2-ae0c-5efed46336c3 admin admin] PATH_INFO: `/v3/users` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:21.525502 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab8818a1-ffae-40f2-ae0c-5efed46336c3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:21.529565 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 894/1810] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:12:21 2022] POST /identity/v3/users => generated 300 bytes in 202 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:12:21.529720 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab8818a1-ffae-40f2-ae0c-5efed46336c3 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:21.531615 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35d42b53-568b-4507-9710-e5099eac3a1c admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:21.544557 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35d42b53-568b-4507-9710-e5099eac3a1c admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:21.556675 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0ef0226-5001-431f-b619-0f10adfdb982 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:21.560223 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0ef0226-5001-431f-b619-0f10adfdb982 admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:21.560518 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0ef0226-5001-431f-b619-0f10adfdb982 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:21.563557 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0ef0226-5001-431f-b619-0f10adfdb982 admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:21.563557 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0ef0226-5001-431f-b619-0f10adfdb982 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:21.564804 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0ef0226-5001-431f-b619-0f10adfdb982 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:21.568402 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 917/1811] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:12:21 2022] POST /identity/v3/users => generated 292 bytes in 311 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:12:21.569466 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab8818a1-ffae-40f2-ae0c-5efed46336c3 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:21.584635 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab8818a1-ffae-40f2-ae0c-5efed46336c3 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:21.585900 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 895/1812] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:12:21 2022] GET /identity/v3/roles => generated 1420 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:12:21.590350 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0ef0226-5001-431f-b619-0f10adfdb982 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:21.598541 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a4ea41b-ac1e-4449-b972-4eee73114dee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:21.599573 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a4ea41b-ac1e-4449-b972-4eee73114dee admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:21.599573 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a4ea41b-ac1e-4449-b972-4eee73114dee admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:21.599821 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a4ea41b-ac1e-4449-b972-4eee73114dee admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:21.600348 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a4ea41b-ac1e-4449-b972-4eee73114dee admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:21.606184 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a4ea41b-ac1e-4449-b972-4eee73114dee admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:21.616798 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0ef0226-5001-431f-b619-0f10adfdb982 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:21.618630 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 918/1813] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:12:21 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:12:21.623328 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a4ea41b-ac1e-4449-b972-4eee73114dee admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:21.626501 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc6df8a5-8ed0-49d2-af31-723a00062898 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:21.627339 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc6df8a5-8ed0-49d2-af31-723a00062898 admin admin] REQUEST_METHOD: `PUT` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:21.627633 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc6df8a5-8ed0-49d2-af31-723a00062898 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:21.627911 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc6df8a5-8ed0-49d2-af31-723a00062898 admin admin] PATH_INFO: `/v3/projects/df7a9f6cfb2c49cb942da08e2c77bf38/users/2ab3d47232094127a61f53ba0d960a6b/roles/9d9d5d2c270146488249b7ed943280db` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:21.642985 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a4ea41b-ac1e-4449-b972-4eee73114dee admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:21.648647 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc6df8a5-8ed0-49d2-af31-723a00062898 admin admin] RBAC: Authorizing `identity:create_grant(project_id=df7a9f6cfb2c49cb942da08e2c77bf38, user_id=2ab3d47232094127a61f53ba0d960a6b, role_id=9d9d5d2c270146488249b7ed943280db)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:21.651796 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc6df8a5-8ed0-49d2-af31-723a00062898 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:21.655409 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6710d593-43c6-4ccc-a1b1-c2257e097bed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:21.655586 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6710d593-43c6-4ccc-a1b1-c2257e097bed admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:21.656081 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6710d593-43c6-4ccc-a1b1-c2257e097bed admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:21.656081 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6710d593-43c6-4ccc-a1b1-c2257e097bed admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:21.656762 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6710d593-43c6-4ccc-a1b1-c2257e097bed admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:21.658507 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6710d593-43c6-4ccc-a1b1-c2257e097bed admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:21.676989 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 919/1814] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:12:21 2022] GET /identity/v3/roles => generated 1420 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 05:12:21.681290 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6710d593-43c6-4ccc-a1b1-c2257e097bed admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:21.685509 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 896/1815] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:12:21 2022] PUT /identity/v3/projects/df7a9f6cfb2c49cb942da08e2c77bf38/users/2ab3d47232094127a61f53ba0d960a6b/roles/9d9d5d2c270146488249b7ed943280db => generated 0 bytes in 96 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:12:21.689436 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc6df8a5-8ed0-49d2-af31-723a00062898 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:21.693544 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6710d593-43c6-4ccc-a1b1-c2257e097bed admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:21.758722 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:21.763800 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc6df8a5-8ed0-49d2-af31-723a00062898 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:21.770929 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64a2553d-b079-4481-b239-c7865d5bad0a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:21.771535 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64a2553d-b079-4481-b239-c7865d5bad0a admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:21.771810 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64a2553d-b079-4481-b239-c7865d5bad0a admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:21.772076 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64a2553d-b079-4481-b239-c7865d5bad0a admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:21.772716 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64a2553d-b079-4481-b239-c7865d5bad0a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:21.774298 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64a2553d-b079-4481-b239-c7865d5bad0a admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:21.784608 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 897/1816] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:12:21 2022] GET /identity/v3/roles => generated 1420 bytes in 96 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:12:21.788587 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c81c1d1f-b94d-4414-9fb0-f64b5e6dc4c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:21.789572 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64a2553d-b079-4481-b239-c7865d5bad0a admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:21.790332 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c81c1d1f-b94d-4414-9fb0-f64b5e6dc4c9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:21.791041 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c81c1d1f-b94d-4414-9fb0-f64b5e6dc4c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:21.791732 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c81c1d1f-b94d-4414-9fb0-f64b5e6dc4c9 admin admin] PATH_INFO: `/v3/projects/e972bb0e3d8f446caf920c6f378df9bf/users/fb10b00acb54452abd27a8920a4965eb/roles/6f03d86980c34c36b44a8d6764ff4fda` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:21.808193 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c81c1d1f-b94d-4414-9fb0-f64b5e6dc4c9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e972bb0e3d8f446caf920c6f378df9bf, user_id=fb10b00acb54452abd27a8920a4965eb, role_id=6f03d86980c34c36b44a8d6764ff4fda)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:21.811166 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c81c1d1f-b94d-4414-9fb0-f64b5e6dc4c9 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:21.811506 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64a2553d-b079-4481-b239-c7865d5bad0a admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:21.814852 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:21.819724 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a37ba7ce-eda2-4077-b18d-3096d9689295 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:21.820470 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a37ba7ce-eda2-4077-b18d-3096d9689295 admin admin] REQUEST_METHOD: `PUT` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:21.820838 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a37ba7ce-eda2-4077-b18d-3096d9689295 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:21.821099 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a37ba7ce-eda2-4077-b18d-3096d9689295 admin admin] PATH_INFO: `/v3/projects/df7a9f6cfb2c49cb942da08e2c77bf38/users/2ab3d47232094127a61f53ba0d960a6b/roles/9d9d5d2c270146488249b7ed943280db` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:21.824227 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a37ba7ce-eda2-4077-b18d-3096d9689295 admin admin] RBAC: Authorizing `identity:create_grant(project_id=df7a9f6cfb2c49cb942da08e2c77bf38, user_id=2ab3d47232094127a61f53ba0d960a6b, role_id=9d9d5d2c270146488249b7ed943280db)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:21.827376 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a37ba7ce-eda2-4077-b18d-3096d9689295 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:21.830804 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 920/1817] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:12:21 2022] PUT /identity/v3/projects/e972bb0e3d8f446caf920c6f378df9bf/users/fb10b00acb54452abd27a8920a4965eb/roles/6f03d86980c34c36b44a8d6764ff4fda => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 05:12:21.842225 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ab96713-a2db-4b7d-b557-4ef254c19ca5 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:21.842807 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ab96713-a2db-4b7d-b557-4ef254c19ca5 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:21.843258 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ab96713-a2db-4b7d-b557-4ef254c19ca5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:21.852116 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 898/1818] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:12:21 2022] PUT /identity/v3/projects/df7a9f6cfb2c49cb942da08e2c77bf38/users/2ab3d47232094127a61f53ba0d960a6b/roles/9d9d5d2c270146488249b7ed943280db => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:12:21.855436 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba8866ff-8ffe-42e8-a958-c0e144f24252 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:21.855819 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba8866ff-8ffe-42e8-a958-c0e144f24252 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:21.856105 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba8866ff-8ffe-42e8-a958-c0e144f24252 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:21.885128 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-9ab96713-a2db-4b7d-b557-4ef254c19ca5 None None] MFA Rules not processed for user `fb10b00acb54452abd27a8920a4965eb`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:12:21.887060 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-9ab96713-a2db-4b7d-b557-4ef254c19ca5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:21.889245 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-ba8866ff-8ffe-42e8-a958-c0e144f24252 None None] MFA Rules not processed for user `2ab3d47232094127a61f53ba0d960a6b`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:12:21.890436 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-ba8866ff-8ffe-42e8-a958-c0e144f24252 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:21.980274 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 921/1819] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:12:21 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 05:12:21.984791 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ab96713-a2db-4b7d-b557-4ef254c19ca5 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:21.997340 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 899/1820] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:12:21 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 05:12:22.011915 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba8866ff-8ffe-42e8-a958-c0e144f24252 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:22.013327 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-ba8866ff-8ffe-42e8-a958-c0e144f24252 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:22.066682 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ab96713-a2db-4b7d-b557-4ef254c19ca5 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:22.071384 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b35cb6b-1dd8-47da-9c67-e0ad0d48dfb7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:22.072010 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b35cb6b-1dd8-47da-9c67-e0ad0d48dfb7 admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:22.072270 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b35cb6b-1dd8-47da-9c67-e0ad0d48dfb7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:22.072537 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b35cb6b-1dd8-47da-9c67-e0ad0d48dfb7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:22.073167 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b35cb6b-1dd8-47da-9c67-e0ad0d48dfb7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:22.075167 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b35cb6b-1dd8-47da-9c67-e0ad0d48dfb7 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:22.080178 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba8866ff-8ffe-42e8-a958-c0e144f24252 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:22.083886 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72bee389-8d36-4bc2-915b-1be867fcaa53 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '019a79b2affe478b8a30d1c538d0415c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:22.084490 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72bee389-8d36-4bc2-915b-1be867fcaa53 service cinder] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:22.084770 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72bee389-8d36-4bc2-915b-1be867fcaa53 service cinder] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:22.085040 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72bee389-8d36-4bc2-915b-1be867fcaa53 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:22.095209 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72bee389-8d36-4bc2-915b-1be867fcaa53 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:22.097213 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72bee389-8d36-4bc2-915b-1be867fcaa53 service cinder] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:22.104257 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 922/1821] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:12:21 2022] POST /identity/v3/projects => generated 383 bytes in 120 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 05:12:22.108106 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b35cb6b-1dd8-47da-9c67-e0ad0d48dfb7 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:22.108521 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 900/1822] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:12:22 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:12:22.199421 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b35cb6b-1dd8-47da-9c67-e0ad0d48dfb7 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:22.216367 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be1f93c0-6a11-417d-83cd-12d299b284c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:22.216993 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be1f93c0-6a11-417d-83cd-12d299b284c7 admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:22.217282 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be1f93c0-6a11-417d-83cd-12d299b284c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:22.217540 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be1f93c0-6a11-417d-83cd-12d299b284c7 admin admin] PATH_INFO: `/v3/users` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:22.218541 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be1f93c0-6a11-417d-83cd-12d299b284c7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:22.224171 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be1f93c0-6a11-417d-83cd-12d299b284c7 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:22.248566 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:22.259969 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 923/1823] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:12:22 2022] POST /identity/v3/users => generated 298 bytes in 152 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:12:22.263839 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72bee389-8d36-4bc2-915b-1be867fcaa53 service cinder] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:22.278999 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72bee389-8d36-4bc2-915b-1be867fcaa53 service cinder] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:22.284184 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be1f93c0-6a11-417d-83cd-12d299b284c7 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:22.286041 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc8c314b-862e-4c8d-9329-29c55fd227ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:22.286927 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc8c314b-862e-4c8d-9329-29c55fd227ec admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:22.287497 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc8c314b-862e-4c8d-9329-29c55fd227ec admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:22.287905 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc8c314b-862e-4c8d-9329-29c55fd227ec admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:22.288689 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc8c314b-862e-4c8d-9329-29c55fd227ec admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:22.290590 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc8c314b-862e-4c8d-9329-29c55fd227ec admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:22.294950 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be1f93c0-6a11-417d-83cd-12d299b284c7 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:22.299643 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a8bb7ed-53eb-445c-bfd4-02e27be33818 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:22.300810 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a8bb7ed-53eb-445c-bfd4-02e27be33818 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:22.301086 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a8bb7ed-53eb-445c-bfd4-02e27be33818 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:22.301354 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a8bb7ed-53eb-445c-bfd4-02e27be33818 admin admin] PATH_INFO: `/v3/users/2cecbe6f7aa043a79ce11446caaea33b` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:22.302458 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 901/1824] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:12:22 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:12:22.302571 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a8bb7ed-53eb-445c-bfd4-02e27be33818 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2cecbe6f7aa043a79ce11446caaea33b)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:22.304389 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a8bb7ed-53eb-445c-bfd4-02e27be33818 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:22.306719 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc8c314b-862e-4c8d-9329-29c55fd227ec admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:22.316785 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc8c314b-862e-4c8d-9329-29c55fd227ec admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:22.321592 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eee646a5-0712-4ed7-b086-cef2cfbf6708 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:22.322232 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee646a5-0712-4ed7-b086-cef2cfbf6708 admin admin] REQUEST_METHOD: `PUT` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:22.322525 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee646a5-0712-4ed7-b086-cef2cfbf6708 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:22.322944 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee646a5-0712-4ed7-b086-cef2cfbf6708 admin admin] PATH_INFO: `/v3/projects/6fb323c928e042eb94b090c8c875af91/users/712f11bcd1644a69858ca0e891faeb3c/roles/9d9d5d2c270146488249b7ed943280db` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:22.341225 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eee646a5-0712-4ed7-b086-cef2cfbf6708 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6fb323c928e042eb94b090c8c875af91, user_id=712f11bcd1644a69858ca0e891faeb3c, role_id=9d9d5d2c270146488249b7ed943280db)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:22.344220 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eee646a5-0712-4ed7-b086-cef2cfbf6708 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:22.354542 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-7a8bb7ed-53eb-445c-bfd4-02e27be33818 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2cecbe6f7aa043a79ce11446caaea33b'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:22.360694 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 902/1825] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:12:22 2022] PUT /identity/v3/projects/6fb323c928e042eb94b090c8c875af91/users/712f11bcd1644a69858ca0e891faeb3c/roles/9d9d5d2c270146488249b7ed943280db => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 05:12:22.363733 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-7a8bb7ed-53eb-445c-bfd4-02e27be33818 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2cecbe6f7aa043a79ce11446caaea33b'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:22.365703 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eee646a5-0712-4ed7-b086-cef2cfbf6708 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:22.386880 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-7a8bb7ed-53eb-445c-bfd4-02e27be33818 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2cecbe6f7aa043a79ce11446caaea33b'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:22.395612 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-7a8bb7ed-53eb-445c-bfd4-02e27be33818 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2cecbe6f7aa043a79ce11446caaea33b'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:22.396353 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-7a8bb7ed-53eb-445c-bfd4-02e27be33818 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2cecbe6f7aa043a79ce11446caaea33b'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:22.406235 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 924/1826] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:12:22 2022] DELETE /identity/v3/users/2cecbe6f7aa043a79ce11446caaea33b => generated 0 bytes in 120 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:12:22.407899 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b77f4ca-40c3-472e-be46-a003b79bf356 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:22.408185 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b77f4ca-40c3-472e-be46-a003b79bf356 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:22.408452 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b77f4ca-40c3-472e-be46-a003b79bf356 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:22.414385 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:22.431744 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-1b77f4ca-40c3-472e-be46-a003b79bf356 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:12:22.432985 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-1b77f4ca-40c3-472e-be46-a003b79bf356 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:22.478546 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eee646a5-0712-4ed7-b086-cef2cfbf6708 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:22.601035 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 925/1827] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:12:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:12:22.611715 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b77f4ca-40c3-472e-be46-a003b79bf356 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:22.614027 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-1b77f4ca-40c3-472e-be46-a003b79bf356 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:22.632702 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b947c6e3-7c2a-4160-a59c-1d8fe9094b05 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:22.633310 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b947c6e3-7c2a-4160-a59c-1d8fe9094b05 admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:22.633585 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b947c6e3-7c2a-4160-a59c-1d8fe9094b05 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:22.633853 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b947c6e3-7c2a-4160-a59c-1d8fe9094b05 admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:22.634495 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b947c6e3-7c2a-4160-a59c-1d8fe9094b05 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:22.636136 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b947c6e3-7c2a-4160-a59c-1d8fe9094b05 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:22.647504 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 903/1828] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:12:22 2022] GET /identity/v3/roles => generated 1420 bytes in 282 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:12:22.652674 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b947c6e3-7c2a-4160-a59c-1d8fe9094b05 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:22.655072 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-b947c6e3-7c2a-4160-a59c-1d8fe9094b05 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:22.686637 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b947c6e3-7c2a-4160-a59c-1d8fe9094b05 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:22.691866 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9a99101-e4de-4e59-a10e-12c30fcf2508 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:22.692524 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a99101-e4de-4e59-a10e-12c30fcf2508 admin admin] REQUEST_METHOD: `PUT` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:22.692803 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a99101-e4de-4e59-a10e-12c30fcf2508 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:22.693075 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a99101-e4de-4e59-a10e-12c30fcf2508 admin admin] PATH_INFO: `/v3/projects/6fb323c928e042eb94b090c8c875af91/users/712f11bcd1644a69858ca0e891faeb3c/roles/9d9d5d2c270146488249b7ed943280db` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:22.695045 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9a99101-e4de-4e59-a10e-12c30fcf2508 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6fb323c928e042eb94b090c8c875af91, user_id=712f11bcd1644a69858ca0e891faeb3c, role_id=9d9d5d2c270146488249b7ed943280db)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:22.697712 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9a99101-e4de-4e59-a10e-12c30fcf2508 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:22.700539 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b77f4ca-40c3-472e-be46-a003b79bf356 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:22.704697 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94a97b3d-5eb0-4381-b5d3-341b8d6a816f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:22.705310 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a97b3d-5eb0-4381-b5d3-341b8d6a816f service neutron] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:22.705591 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a97b3d-5eb0-4381-b5d3-341b8d6a816f service neutron] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:22.705861 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a97b3d-5eb0-4381-b5d3-341b8d6a816f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:22.710172 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 904/1829] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:12:22 2022] PUT /identity/v3/projects/6fb323c928e042eb94b090c8c875af91/users/712f11bcd1644a69858ca0e891faeb3c/roles/9d9d5d2c270146488249b7ed943280db => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:12:22.719473 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94a97b3d-5eb0-4381-b5d3-341b8d6a816f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:22.721822 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94a97b3d-5eb0-4381-b5d3-341b8d6a816f service neutron] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:22.722925 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-870d16b5-9aca-45ce-ae9f-f6616e4eebb9 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:22.723396 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-870d16b5-9aca-45ce-ae9f-f6616e4eebb9 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:22.724003 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-870d16b5-9aca-45ce-ae9f-f6616e4eebb9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:22.734370 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 926/1830] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:12:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 124 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:12:22.767506 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-870d16b5-9aca-45ce-ae9f-f6616e4eebb9 None None] MFA Rules not processed for user `712f11bcd1644a69858ca0e891faeb3c`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:12:22.768737 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-870d16b5-9aca-45ce-ae9f-f6616e4eebb9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:22.926705 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 905/1831] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:12:22 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 206 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:12:22.943146 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94a97b3d-5eb0-4381-b5d3-341b8d6a816f service neutron] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:22.945284 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-94a97b3d-5eb0-4381-b5d3-341b8d6a816f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:22.985722 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:23.041791 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94a97b3d-5eb0-4381-b5d3-341b8d6a816f service neutron] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:23.101304 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5e0f0ad-b6ed-47a5-acba-28009ef7a5b3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '019a79b2affe478b8a30d1c538d0415c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:23.101946 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5e0f0ad-b6ed-47a5-acba-28009ef7a5b3 service cinder] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:23.102226 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5e0f0ad-b6ed-47a5-acba-28009ef7a5b3 service cinder] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:23.102493 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5e0f0ad-b6ed-47a5-acba-28009ef7a5b3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:23.104328 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-a5e0f0ad-b6ed-47a5-acba-28009ef7a5b3 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:23.138623 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-870d16b5-9aca-45ce-ae9f-f6616e4eebb9 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:23.140208 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-870d16b5-9aca-45ce-ae9f-f6616e4eebb9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:23.197327 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5e0f0ad-b6ed-47a5-acba-28009ef7a5b3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:23.199235 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5e0f0ad-b6ed-47a5-acba-28009ef7a5b3 service cinder] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:23.212429 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 927/1832] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:12:22 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 270 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:12:23.272730 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:23.299717 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-870d16b5-9aca-45ce-ae9f-f6616e4eebb9 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:23.305952 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73f3852b-8a5b-4ef1-a382-48719871833e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:23.306677 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73f3852b-8a5b-4ef1-a382-48719871833e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:23.307062 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73f3852b-8a5b-4ef1-a382-48719871833e admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:23.307431 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73f3852b-8a5b-4ef1-a382-48719871833e admin admin] PATH_INFO: `/v3/projects/0f41c6bcddee420791540051817ac6d5` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:23.308750 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73f3852b-8a5b-4ef1-a382-48719871833e admin admin] RBAC: Authorizing `identity:delete_project(project_id=0f41c6bcddee420791540051817ac6d5)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:23.311124 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73f3852b-8a5b-4ef1-a382-48719871833e admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:23.396232 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-73f3852b-8a5b-4ef1-a382-48719871833e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0f41c6bcddee420791540051817ac6d5', 'request_id': 'req-73f3852b-8a5b-4ef1-a382-48719871833e'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:23.416289 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-73f3852b-8a5b-4ef1-a382-48719871833e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0f41c6bcddee420791540051817ac6d5', 'request_id': 'req-73f3852b-8a5b-4ef1-a382-48719871833e'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:23.436663 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-73f3852b-8a5b-4ef1-a382-48719871833e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0f41c6bcddee420791540051817ac6d5', 'request_id': 'req-73f3852b-8a5b-4ef1-a382-48719871833e'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:23.457565 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-73f3852b-8a5b-4ef1-a382-48719871833e admin admin] The token cache is being invalidate because project 0f41c6bcddee420791540051817ac6d5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81624) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:12:23.458403 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-73f3852b-8a5b-4ef1-a382-48719871833e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:23.459688 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-73f3852b-8a5b-4ef1-a382-48719871833e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:23.462427 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 906/1833] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:12:23 2022] DELETE /identity/v3/projects/0f41c6bcddee420791540051817ac6d5 => generated 0 bytes in 324 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 05:12:23.466440 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5e0f0ad-b6ed-47a5-acba-28009ef7a5b3 service cinder] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:23.486164 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5e0f0ad-b6ed-47a5-acba-28009ef7a5b3 service cinder] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:23.624046 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb717bb7-1ab0-4f5b-90cd-19e345d27e4a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:23.624875 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb717bb7-1ab0-4f5b-90cd-19e345d27e4a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:23.625260 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb717bb7-1ab0-4f5b-90cd-19e345d27e4a admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:23.625636 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb717bb7-1ab0-4f5b-90cd-19e345d27e4a admin admin] PATH_INFO: `/v3/users/2ab3d47232094127a61f53ba0d960a6b` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:23.626979 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb717bb7-1ab0-4f5b-90cd-19e345d27e4a admin admin] RBAC: Authorizing `identity:delete_user(user_id=2ab3d47232094127a61f53ba0d960a6b)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:23.629007 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb717bb7-1ab0-4f5b-90cd-19e345d27e4a admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:23.703803 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:23.751579 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-eb717bb7-1ab0-4f5b-90cd-19e345d27e4a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2ab3d47232094127a61f53ba0d960a6b'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:23.773781 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-eb717bb7-1ab0-4f5b-90cd-19e345d27e4a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2ab3d47232094127a61f53ba0d960a6b'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:23.833045 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-eb717bb7-1ab0-4f5b-90cd-19e345d27e4a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2ab3d47232094127a61f53ba0d960a6b'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:23.842301 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-eb717bb7-1ab0-4f5b-90cd-19e345d27e4a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2ab3d47232094127a61f53ba0d960a6b'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:23.843143 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-eb717bb7-1ab0-4f5b-90cd-19e345d27e4a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2ab3d47232094127a61f53ba0d960a6b'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:23.853015 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 928/1834] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:12:23 2022] DELETE /identity/v3/users/2ab3d47232094127a61f53ba0d960a6b => generated 0 bytes in 387 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:12:24.174932 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73f3852b-8a5b-4ef1-a382-48719871833e admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:24.176917 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-73f3852b-8a5b-4ef1-a382-48719871833e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:24.190502 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73f3852b-8a5b-4ef1-a382-48719871833e admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:24.248129 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32179c8e-68dc-4d28-8df6-9248c4d33574 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:24.248700 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32179c8e-68dc-4d28-8df6-9248c4d33574 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:24.248973 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32179c8e-68dc-4d28-8df6-9248c4d33574 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:24.249237 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32179c8e-68dc-4d28-8df6-9248c4d33574 admin admin] PATH_INFO: `/v3/projects/df7a9f6cfb2c49cb942da08e2c77bf38` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:24.250327 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32179c8e-68dc-4d28-8df6-9248c4d33574 admin admin] RBAC: Authorizing `identity:delete_project(project_id=df7a9f6cfb2c49cb942da08e2c77bf38)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:24.252239 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32179c8e-68dc-4d28-8df6-9248c4d33574 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:24.296796 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-32179c8e-68dc-4d28-8df6-9248c4d33574 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'df7a9f6cfb2c49cb942da08e2c77bf38', 'request_id': 'req-32179c8e-68dc-4d28-8df6-9248c4d33574'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:24.304413 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-32179c8e-68dc-4d28-8df6-9248c4d33574 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'df7a9f6cfb2c49cb942da08e2c77bf38', 'request_id': 'req-32179c8e-68dc-4d28-8df6-9248c4d33574'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:24.310474 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-32179c8e-68dc-4d28-8df6-9248c4d33574 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'df7a9f6cfb2c49cb942da08e2c77bf38', 'request_id': 'req-32179c8e-68dc-4d28-8df6-9248c4d33574'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:24.321149 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-32179c8e-68dc-4d28-8df6-9248c4d33574 admin admin] The token cache is being invalidate because project df7a9f6cfb2c49cb942da08e2c77bf38 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81624) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:12:24.321498 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-32179c8e-68dc-4d28-8df6-9248c4d33574 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:24.322119 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-32179c8e-68dc-4d28-8df6-9248c4d33574 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:24.323603 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 907/1835] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:12:24 2022] DELETE /identity/v3/projects/df7a9f6cfb2c49cb942da08e2c77bf38 => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:12:24.328828 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3d5998e-27ec-420b-8821-65128ff47242 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:24.329268 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3d5998e-27ec-420b-8821-65128ff47242 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:24.329501 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3d5998e-27ec-420b-8821-65128ff47242 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:24.350274 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-b3d5998e-27ec-420b-8821-65128ff47242 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:12:24.351509 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-b3d5998e-27ec-420b-8821-65128ff47242 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:24.410655 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 929/1836] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:12:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:12:24.418899 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e33c203-e44c-4ee7-a3c2-388dd9ef204b None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:24.419185 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e33c203-e44c-4ee7-a3c2-388dd9ef204b None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:24.419456 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e33c203-e44c-4ee7-a3c2-388dd9ef204b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:24.435033 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:24.441310 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-1e33c203-e44c-4ee7-a3c2-388dd9ef204b None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:12:24.442805 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-1e33c203-e44c-4ee7-a3c2-388dd9ef204b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:24.451950 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 908/1837] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:12:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:12:24.456258 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3d5998e-27ec-420b-8821-65128ff47242 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:24.466255 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3d5998e-27ec-420b-8821-65128ff47242 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:24.471746 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40dcd6f0-a7a7-46fe-ab22-174b5ffd4613 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:24.472823 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40dcd6f0-a7a7-46fe-ab22-174b5ffd4613 admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:24.473102 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40dcd6f0-a7a7-46fe-ab22-174b5ffd4613 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:24.473369 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40dcd6f0-a7a7-46fe-ab22-174b5ffd4613 admin admin] PATH_INFO: `/v3/domains` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:24.474007 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40dcd6f0-a7a7-46fe-ab22-174b5ffd4613 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:12:24.474356 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40dcd6f0-a7a7-46fe-ab22-174b5ffd4613 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:24.475961 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40dcd6f0-a7a7-46fe-ab22-174b5ffd4613 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:24.488526 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 930/1838] 10.222.0.77 () {58 vars in 1153 bytes} [Tue Aug 30 05:12:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 05:12:24.493041 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e33c203-e44c-4ee7-a3c2-388dd9ef204b None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:24.503065 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e33c203-e44c-4ee7-a3c2-388dd9ef204b None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:24.507644 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d17c24f9-a210-4e46-9c6b-612d1569c685 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:24.508231 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d17c24f9-a210-4e46-9c6b-612d1569c685 admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:24.508507 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d17c24f9-a210-4e46-9c6b-612d1569c685 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:24.508777 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d17c24f9-a210-4e46-9c6b-612d1569c685 admin admin] PATH_INFO: `/v3/projects` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:24.509401 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d17c24f9-a210-4e46-9c6b-612d1569c685 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:24.511313 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d17c24f9-a210-4e46-9c6b-612d1569c685 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:24.537134 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 909/1839] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:12:24 2022] POST /identity/v3/projects => generated 379 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:12:24.541182 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40dcd6f0-a7a7-46fe-ab22-174b5ffd4613 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:24.551981 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40dcd6f0-a7a7-46fe-ab22-174b5ffd4613 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:24.607108 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3171cc6f-8017-403f-a054-311215d672c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:24.607726 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3171cc6f-8017-403f-a054-311215d672c6 admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:24.608001 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3171cc6f-8017-403f-a054-311215d672c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:24.608271 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3171cc6f-8017-403f-a054-311215d672c6 admin admin] PATH_INFO: `/v3/users` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:24.608894 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3171cc6f-8017-403f-a054-311215d672c6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:24.610794 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3171cc6f-8017-403f-a054-311215d672c6 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:24.637054 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 931/1840] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:12:24 2022] POST /identity/v3/users => generated 290 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 05:12:24.638307 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d17c24f9-a210-4e46-9c6b-612d1569c685 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:24.648479 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d17c24f9-a210-4e46-9c6b-612d1569c685 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:24.652912 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08ddf423-afea-4185-a360-33c061c07a7f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:24.653492 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08ddf423-afea-4185-a360-33c061c07a7f admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:24.653770 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08ddf423-afea-4185-a360-33c061c07a7f admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:24.654035 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08ddf423-afea-4185-a360-33c061c07a7f admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:24.654662 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08ddf423-afea-4185-a360-33c061c07a7f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:24.656290 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08ddf423-afea-4185-a360-33c061c07a7f admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:24.670807 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 910/1841] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:12:24 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:12:24.671744 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3171cc6f-8017-403f-a054-311215d672c6 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:24.682708 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3171cc6f-8017-403f-a054-311215d672c6 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:24.687293 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a4aa392-b591-4c32-b957-3ed0712e2622 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:24.687885 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a4aa392-b591-4c32-b957-3ed0712e2622 admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:24.688163 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a4aa392-b591-4c32-b957-3ed0712e2622 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:24.688435 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a4aa392-b591-4c32-b957-3ed0712e2622 admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:24.689070 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a4aa392-b591-4c32-b957-3ed0712e2622 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:24.690677 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a4aa392-b591-4c32-b957-3ed0712e2622 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:24.701368 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 932/1842] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:12:24 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:12:24.705509 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08ddf423-afea-4185-a360-33c061c07a7f admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:24.716066 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08ddf423-afea-4185-a360-33c061c07a7f admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:24.723225 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e77b8c5-52ef-4c70-8168-74410b4a30b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:24.723890 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e77b8c5-52ef-4c70-8168-74410b4a30b9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:24.724162 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e77b8c5-52ef-4c70-8168-74410b4a30b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:24.724425 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e77b8c5-52ef-4c70-8168-74410b4a30b9 admin admin] PATH_INFO: `/v3/projects/bec16540a4be4e40bf3bab0ef22861b5/users/b5a78e5138e941b586335f796244f7fc/roles/6f03d86980c34c36b44a8d6764ff4fda` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:24.744167 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e77b8c5-52ef-4c70-8168-74410b4a30b9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bec16540a4be4e40bf3bab0ef22861b5, user_id=b5a78e5138e941b586335f796244f7fc, role_id=6f03d86980c34c36b44a8d6764ff4fda)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:24.747113 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e77b8c5-52ef-4c70-8168-74410b4a30b9 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:24.772498 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 911/1843] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:12:24 2022] PUT /identity/v3/projects/bec16540a4be4e40bf3bab0ef22861b5/users/b5a78e5138e941b586335f796244f7fc/roles/6f03d86980c34c36b44a8d6764ff4fda => generated 0 bytes in 67 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 05:12:24.783258 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df52792-8449-4ac0-8f93-47661d5567b1 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:24.783545 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df52792-8449-4ac0-8f93-47661d5567b1 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:24.783821 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df52792-8449-4ac0-8f93-47661d5567b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:24.805967 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=91,keystone:DELETE=5,keystone:INSERT=4 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:24.815180 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-5df52792-8449-4ac0-8f93-47661d5567b1 None None] MFA Rules not processed for user `b5a78e5138e941b586335f796244f7fc`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:12:24.816453 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-5df52792-8449-4ac0-8f93-47661d5567b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:24.886942 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 933/1844] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:12:24 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:12:24.891026 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e77b8c5-52ef-4c70-8168-74410b4a30b9 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:24.961327 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e77b8c5-52ef-4c70-8168-74410b4a30b9 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:24.966101 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-478c6435-5f92-45aa-9d62-ea35c7ee7b45 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:24.966930 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478c6435-5f92-45aa-9d62-ea35c7ee7b45 admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:24.967176 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478c6435-5f92-45aa-9d62-ea35c7ee7b45 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:24.967563 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478c6435-5f92-45aa-9d62-ea35c7ee7b45 admin admin] PATH_INFO: `/v3/projects` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:24.968487 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-478c6435-5f92-45aa-9d62-ea35c7ee7b45 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:24.971124 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-478c6435-5f92-45aa-9d62-ea35c7ee7b45 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:25.003358 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 912/1845] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:12:24 2022] POST /identity/v3/projects => generated 379 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 05:12:25.004723 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5df52792-8449-4ac0-8f93-47661d5567b1 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:25.016487 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5df52792-8449-4ac0-8f93-47661d5567b1 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:25.076767 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b355e2f-9cae-4082-9f53-750eff12b8d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:25.077364 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b355e2f-9cae-4082-9f53-750eff12b8d4 admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:25.077646 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b355e2f-9cae-4082-9f53-750eff12b8d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:25.078293 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b355e2f-9cae-4082-9f53-750eff12b8d4 admin admin] PATH_INFO: `/v3/users` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:25.079006 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b355e2f-9cae-4082-9f53-750eff12b8d4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:25.080929 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b355e2f-9cae-4082-9f53-750eff12b8d4 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:25.103560 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 934/1846] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:12:25 2022] POST /identity/v3/users => generated 290 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:12:25.107477 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-478c6435-5f92-45aa-9d62-ea35c7ee7b45 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:25.117720 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-478c6435-5f92-45aa-9d62-ea35c7ee7b45 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:25.122315 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61bec7ac-f4b4-44b5-a140-c0cd6a33765b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:25.123285 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61bec7ac-f4b4-44b5-a140-c0cd6a33765b admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:25.123560 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61bec7ac-f4b4-44b5-a140-c0cd6a33765b admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:25.123852 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61bec7ac-f4b4-44b5-a140-c0cd6a33765b admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:25.124479 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61bec7ac-f4b4-44b5-a140-c0cd6a33765b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:25.126047 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61bec7ac-f4b4-44b5-a140-c0cd6a33765b admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:25.136623 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 913/1847] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:12:25 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:12:25.140594 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b355e2f-9cae-4082-9f53-750eff12b8d4 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:25.150722 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b355e2f-9cae-4082-9f53-750eff12b8d4 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:25.155338 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdee378a-0998-4e80-834d-8fd3ae98d92b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:25.155922 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdee378a-0998-4e80-834d-8fd3ae98d92b admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:25.156198 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdee378a-0998-4e80-834d-8fd3ae98d92b admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:25.156465 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdee378a-0998-4e80-834d-8fd3ae98d92b admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:25.157122 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdee378a-0998-4e80-834d-8fd3ae98d92b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:25.158739 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdee378a-0998-4e80-834d-8fd3ae98d92b admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:25.169018 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 935/1848] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:12:25 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 05:12:25.172944 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61bec7ac-f4b4-44b5-a140-c0cd6a33765b admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:25.182757 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61bec7ac-f4b4-44b5-a140-c0cd6a33765b admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:25.187343 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e186c01b-f1e9-4f43-bddf-1cca5574e900 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:25.187966 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e186c01b-f1e9-4f43-bddf-1cca5574e900 admin admin] REQUEST_METHOD: `PUT` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:25.188244 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e186c01b-f1e9-4f43-bddf-1cca5574e900 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:25.188534 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e186c01b-f1e9-4f43-bddf-1cca5574e900 admin admin] PATH_INFO: `/v3/projects/fbc9ec8aa1f742eebc6da392cfafd613/users/2abfa5603b39423d9c386bda4bc22603/roles/6f03d86980c34c36b44a8d6764ff4fda` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:25.202545 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e186c01b-f1e9-4f43-bddf-1cca5574e900 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fbc9ec8aa1f742eebc6da392cfafd613, user_id=2abfa5603b39423d9c386bda4bc22603, role_id=6f03d86980c34c36b44a8d6764ff4fda)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:25.205199 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e186c01b-f1e9-4f43-bddf-1cca5574e900 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:25.218380 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 914/1849] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:12:25 2022] PUT /identity/v3/projects/fbc9ec8aa1f742eebc6da392cfafd613/users/2abfa5603b39423d9c386bda4bc22603/roles/6f03d86980c34c36b44a8d6764ff4fda => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:12:25.230912 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52ae0746-beb0-4230-b31d-25d54b14fca7 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:25.231205 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52ae0746-beb0-4230-b31d-25d54b14fca7 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:25.231479 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52ae0746-beb0-4230-b31d-25d54b14fca7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:25.259990 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-52ae0746-beb0-4230-b31d-25d54b14fca7 None None] MFA Rules not processed for user `2abfa5603b39423d9c386bda4bc22603`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:12:25.261236 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-52ae0746-beb0-4230-b31d-25d54b14fca7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:25.324470 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 936/1850] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:12:25 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:12:25.328978 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e186c01b-f1e9-4f43-bddf-1cca5574e900 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:25.380137 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:25.390997 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e186c01b-f1e9-4f43-bddf-1cca5574e900 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:25.396183 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7837bb65-261f-46e2-bcec-6f0af6742e46 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:25.396779 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7837bb65-261f-46e2-bcec-6f0af6742e46 admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:25.397056 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7837bb65-261f-46e2-bcec-6f0af6742e46 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:25.397324 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7837bb65-261f-46e2-bcec-6f0af6742e46 admin admin] PATH_INFO: `/v3/projects` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:25.397950 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7837bb65-261f-46e2-bcec-6f0af6742e46 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:25.399869 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7837bb65-261f-46e2-bcec-6f0af6742e46 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:25.425197 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 915/1851] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:12:25 2022] POST /identity/v3/projects => generated 379 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:12:25.429334 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52ae0746-beb0-4230-b31d-25d54b14fca7 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:25.440484 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52ae0746-beb0-4230-b31d-25d54b14fca7 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:25.445479 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:25.499519 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d07cf3db-8c1f-4d7d-9338-a729216edc0b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:25.500131 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d07cf3db-8c1f-4d7d-9338-a729216edc0b admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:25.500410 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d07cf3db-8c1f-4d7d-9338-a729216edc0b admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:25.500681 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d07cf3db-8c1f-4d7d-9338-a729216edc0b admin admin] PATH_INFO: `/v3/users` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:25.501317 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d07cf3db-8c1f-4d7d-9338-a729216edc0b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:25.503854 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d07cf3db-8c1f-4d7d-9338-a729216edc0b admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:25.524953 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 937/1852] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:12:25 2022] POST /identity/v3/users => generated 296 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:12:25.529016 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7837bb65-261f-46e2-bcec-6f0af6742e46 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:25.539380 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7837bb65-261f-46e2-bcec-6f0af6742e46 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:25.543856 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b510516b-8871-4d28-b0ed-446c76b2634d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:25.544444 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b510516b-8871-4d28-b0ed-446c76b2634d admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:25.544722 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b510516b-8871-4d28-b0ed-446c76b2634d admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:25.544990 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b510516b-8871-4d28-b0ed-446c76b2634d admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:25.545614 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b510516b-8871-4d28-b0ed-446c76b2634d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:25.547263 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b510516b-8871-4d28-b0ed-446c76b2634d admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:25.557584 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 916/1853] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:12:25 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:12:25.561576 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d07cf3db-8c1f-4d7d-9338-a729216edc0b admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:25.571838 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d07cf3db-8c1f-4d7d-9338-a729216edc0b admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:25.576364 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-738cbe42-786f-4e53-816c-d6ee1131b488 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:25.576937 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738cbe42-786f-4e53-816c-d6ee1131b488 admin admin] REQUEST_METHOD: `PUT` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:25.577212 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738cbe42-786f-4e53-816c-d6ee1131b488 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:25.577471 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738cbe42-786f-4e53-816c-d6ee1131b488 admin admin] PATH_INFO: `/v3/projects/1d204a50ea3440de85d9e216ca404ac9/users/c6952bfdf26440058f1391958081a8ed/roles/9d9d5d2c270146488249b7ed943280db` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:25.592397 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-738cbe42-786f-4e53-816c-d6ee1131b488 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1d204a50ea3440de85d9e216ca404ac9, user_id=c6952bfdf26440058f1391958081a8ed, role_id=9d9d5d2c270146488249b7ed943280db)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:25.595308 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-738cbe42-786f-4e53-816c-d6ee1131b488 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:25.608316 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 938/1854] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:12:25 2022] PUT /identity/v3/projects/1d204a50ea3440de85d9e216ca404ac9/users/c6952bfdf26440058f1391958081a8ed/roles/9d9d5d2c270146488249b7ed943280db => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:12:25.612202 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b510516b-8871-4d28-b0ed-446c76b2634d admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:25.622510 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b510516b-8871-4d28-b0ed-446c76b2634d admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:25.683607 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb209429-ae94-45b2-aa81-aa30d8de85b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:25.684213 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb209429-ae94-45b2-aa81-aa30d8de85b5 admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:25.684492 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb209429-ae94-45b2-aa81-aa30d8de85b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:25.684764 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb209429-ae94-45b2-aa81-aa30d8de85b5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:25.685390 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb209429-ae94-45b2-aa81-aa30d8de85b5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:25.687006 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb209429-ae94-45b2-aa81-aa30d8de85b5 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:25.697255 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 917/1855] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:12:25 2022] GET /identity/v3/roles => generated 1420 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:12:25.701360 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-738cbe42-786f-4e53-816c-d6ee1131b488 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:25.711562 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-738cbe42-786f-4e53-816c-d6ee1131b488 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:25.716163 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97f9cfce-6174-4801-b7a9-b654a5545af2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:25.716809 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97f9cfce-6174-4801-b7a9-b654a5545af2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:25.717092 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97f9cfce-6174-4801-b7a9-b654a5545af2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:25.717368 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97f9cfce-6174-4801-b7a9-b654a5545af2 admin admin] PATH_INFO: `/v3/projects/1d204a50ea3440de85d9e216ca404ac9/users/c6952bfdf26440058f1391958081a8ed/roles/9d9d5d2c270146488249b7ed943280db` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:25.719284 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97f9cfce-6174-4801-b7a9-b654a5545af2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1d204a50ea3440de85d9e216ca404ac9, user_id=c6952bfdf26440058f1391958081a8ed, role_id=9d9d5d2c270146488249b7ed943280db)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:25.722213 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97f9cfce-6174-4801-b7a9-b654a5545af2 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:25.732216 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 939/1856] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:12:25 2022] PUT /identity/v3/projects/1d204a50ea3440de85d9e216ca404ac9/users/c6952bfdf26440058f1391958081a8ed/roles/9d9d5d2c270146488249b7ed943280db => generated 0 bytes in 31 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:12:25.742574 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed681ac-a07a-40cb-abba-8aae85f588be None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:25.742925 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed681ac-a07a-40cb-abba-8aae85f588be None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:25.743270 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed681ac-a07a-40cb-abba-8aae85f588be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:25.774249 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-2ed681ac-a07a-40cb-abba-8aae85f588be None None] MFA Rules not processed for user `c6952bfdf26440058f1391958081a8ed`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:12:25.775876 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-2ed681ac-a07a-40cb-abba-8aae85f588be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:25.913292 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 918/1857] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:12:25 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:12:25.925723 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97f9cfce-6174-4801-b7a9-b654a5545af2 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:25.928098 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-97f9cfce-6174-4801-b7a9-b654a5545af2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:26.067749 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97f9cfce-6174-4801-b7a9-b654a5545af2 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:26.082426 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fe46cb6-4582-471a-93af-2fd24015495c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '019a79b2affe478b8a30d1c538d0415c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:26.083344 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe46cb6-4582-471a-93af-2fd24015495c service cinder] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:26.083903 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe46cb6-4582-471a-93af-2fd24015495c service cinder] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:26.085219 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe46cb6-4582-471a-93af-2fd24015495c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:26.109743 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fe46cb6-4582-471a-93af-2fd24015495c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:26.111792 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fe46cb6-4582-471a-93af-2fd24015495c service cinder] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:26.134254 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 940/1858] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:12:25 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 210 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:12:26.230845 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ed681ac-a07a-40cb-abba-8aae85f588be None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:26.232612 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-2ed681ac-a07a-40cb-abba-8aae85f588be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:26.246897 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ed681ac-a07a-40cb-abba-8aae85f588be None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:26.253185 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-451a9bb3-6f20-4b8a-ab99-86a6e62ecb8b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '019a79b2affe478b8a30d1c538d0415c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:26.253804 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-451a9bb3-6f20-4b8a-ab99-86a6e62ecb8b service cinder] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:26.254083 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-451a9bb3-6f20-4b8a-ab99-86a6e62ecb8b service cinder] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:26.254359 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-451a9bb3-6f20-4b8a-ab99-86a6e62ecb8b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:26.321408 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:26.334986 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-451a9bb3-6f20-4b8a-ab99-86a6e62ecb8b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:26.343431 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-451a9bb3-6f20-4b8a-ab99-86a6e62ecb8b service cinder] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:26.355854 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 919/1859] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:12:26 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 125 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:12:28.877627 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-163d8a87-c6a6-4576-8dd4-4cf03eaa62ac None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:28.878162 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-163d8a87-c6a6-4576-8dd4-4cf03eaa62ac None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:28.878910 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-163d8a87-c6a6-4576-8dd4-4cf03eaa62ac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:28.901060 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-163d8a87-c6a6-4576-8dd4-4cf03eaa62ac None None] MFA Rules not processed for user `c33c45f4770d4151858e1d897d09af69`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:12:28.902257 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-163d8a87-c6a6-4576-8dd4-4cf03eaa62ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:28.976308 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 941/1860] 10.222.0.82 () {58 vars in 1015 bytes} [Tue Aug 30 05:12:28 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:12:29.000895 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-451a9bb3-6f20-4b8a-ab99-86a6e62ecb8b service cinder] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:29.001894 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-163d8a87-c6a6-4576-8dd4-4cf03eaa62ac None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:29.002921 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-451a9bb3-6f20-4b8a-ab99-86a6e62ecb8b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:29.003713 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-163d8a87-c6a6-4576-8dd4-4cf03eaa62ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:29.016667 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-451a9bb3-6f20-4b8a-ab99-86a6e62ecb8b service cinder] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:29.017250 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-163d8a87-c6a6-4576-8dd4-4cf03eaa62ac None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:29.022210 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3edbc07-8fb4-467e-8e43-764716dd2964 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:29.023406 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3edbc07-8fb4-467e-8e43-764716dd2964 service neutron] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:29.023692 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3edbc07-8fb4-467e-8e43-764716dd2964 service neutron] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:29.024139 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3edbc07-8fb4-467e-8e43-764716dd2964 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:29.026060 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5271113b-8c54-49c3-8242-13aa4e790a1b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:29.026683 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5271113b-8c54-49c3-8242-13aa4e790a1b service neutron] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:29.026997 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5271113b-8c54-49c3-8242-13aa4e790a1b service neutron] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:29.027270 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5271113b-8c54-49c3-8242-13aa4e790a1b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:29.033699 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3edbc07-8fb4-467e-8e43-764716dd2964 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:29.035812 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3edbc07-8fb4-467e-8e43-764716dd2964 service neutron] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:29.037521 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5271113b-8c54-49c3-8242-13aa4e790a1b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:29.039203 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5271113b-8c54-49c3-8242-13aa4e790a1b service neutron] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:29.046117 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 920/1861] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:12:28 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 05:12:29.053085 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:29.067539 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 942/1862] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:12:29 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 66 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:12:29.435989 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3edbc07-8fb4-467e-8e43-764716dd2964 service neutron] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:29.454869 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3edbc07-8fb4-467e-8e43-764716dd2964 service neutron] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:29.463008 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1191fdc3-6f29-4335-a59d-cdf431cc75de service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '019a79b2affe478b8a30d1c538d0415c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:29.463901 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1191fdc3-6f29-4335-a59d-cdf431cc75de service cinder] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:29.464361 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1191fdc3-6f29-4335-a59d-cdf431cc75de service cinder] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:29.464769 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1191fdc3-6f29-4335-a59d-cdf431cc75de service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:29.470447 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-1191fdc3-6f29-4335-a59d-cdf431cc75de service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:29.541044 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1191fdc3-6f29-4335-a59d-cdf431cc75de service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:29.542769 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1191fdc3-6f29-4335-a59d-cdf431cc75de service cinder] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:29.553916 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 921/1863] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:12:29 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 119 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:12:30.036935 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c1a89c9-2b4c-4bdf-8686-1775c9c485be None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:30.037546 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c1a89c9-2b4c-4bdf-8686-1775c9c485be None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:30.037546 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c1a89c9-2b4c-4bdf-8686-1775c9c485be None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:30.039288 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 943/1864] 10.222.0.77 () {54 vars in 921 bytes} [Tue Aug 30 05:12:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:12:30.046328 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-597953c2-f8f2-43de-b6be-94e8fd02fde4 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:30.046918 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-597953c2-f8f2-43de-b6be-94e8fd02fde4 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:30.047226 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-597953c2-f8f2-43de-b6be-94e8fd02fde4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:30.073490 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-597953c2-f8f2-43de-b6be-94e8fd02fde4 None None] MFA Rules not processed for user `886eb5e00bbf4ec292eb962b665b37d0`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:12:30.074902 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-597953c2-f8f2-43de-b6be-94e8fd02fde4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:30.160365 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 922/1865] 10.222.0.77 () {58 vars in 1018 bytes} [Tue Aug 30 05:12:30 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:12:30.178053 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c1a89c9-2b4c-4bdf-8686-1775c9c485be None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:30.179807 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-2c1a89c9-2b4c-4bdf-8686-1775c9c485be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:30.182896 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94af850f-e9d7-4a8d-a2ce-a777aef69f7d None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:30.183455 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94af850f-e9d7-4a8d-a2ce-a777aef69f7d None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:30.183922 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94af850f-e9d7-4a8d-a2ce-a777aef69f7d None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:30.185801 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 923/1866] 10.222.0.77 () {54 vars in 921 bytes} [Tue Aug 30 05:12:30 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 05:12:30.201120 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c1a89c9-2b4c-4bdf-8686-1775c9c485be None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:30.207434 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b620ea80-2533-4f47-9083-8bbb4708f385 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:30.207961 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b620ea80-2533-4f47-9083-8bbb4708f385 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:30.208525 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b620ea80-2533-4f47-9083-8bbb4708f385 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:30.208819 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cda6102e-a5f7-4847-8cc7-150644848e4a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '886eb5e00bbf4ec292eb962b665b37d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:30.209956 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cda6102e-a5f7-4847-8cc7-150644848e4a service nova] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:30.210269 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cda6102e-a5f7-4847-8cc7-150644848e4a service nova] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:30.210531 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cda6102e-a5f7-4847-8cc7-150644848e4a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:30.225337 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cda6102e-a5f7-4847-8cc7-150644848e4a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:30.226983 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cda6102e-a5f7-4847-8cc7-150644848e4a service nova] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:30.239707 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-b620ea80-2533-4f47-9083-8bbb4708f385 None None] MFA Rules not processed for user `886eb5e00bbf4ec292eb962b665b37d0`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:12:30.241317 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-b620ea80-2533-4f47-9083-8bbb4708f385 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:30.249154 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 944/1867] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:12:30 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 72 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:12:30.256808 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 924/1868] 10.222.0.77 () {58 vars in 1018 bytes} [Tue Aug 30 05:12:30 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 51 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 05:12:30.270227 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cda6102e-a5f7-4847-8cc7-150644848e4a service nova] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:30.272560 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-cda6102e-a5f7-4847-8cc7-150644848e4a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:30.290795 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cda6102e-a5f7-4847-8cc7-150644848e4a service nova] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:30.299470 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a97d951-8222-4d68-88e2-2f5279a0d58f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '886eb5e00bbf4ec292eb962b665b37d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:30.300863 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a97d951-8222-4d68-88e2-2f5279a0d58f service nova] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:30.301224 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a97d951-8222-4d68-88e2-2f5279a0d58f service nova] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:30.301559 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a97d951-8222-4d68-88e2-2f5279a0d58f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:30.316828 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a97d951-8222-4d68-88e2-2f5279a0d58f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:30.318577 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a97d951-8222-4d68-88e2-2f5279a0d58f service nova] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:30.334199 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 945/1869] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:12:30 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 64 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:12:30.572978 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b620ea80-2533-4f47-9083-8bbb4708f385 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:30.582140 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b620ea80-2533-4f47-9083-8bbb4708f385 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:30.585614 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-495743e2-b660-4b91-b1bf-db77851c2f71 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '019a79b2affe478b8a30d1c538d0415c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:30.586203 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-495743e2-b660-4b91-b1bf-db77851c2f71 service cinder] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:30.586483 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-495743e2-b660-4b91-b1bf-db77851c2f71 service cinder] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:30.586753 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-495743e2-b660-4b91-b1bf-db77851c2f71 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:30.588535 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-495743e2-b660-4b91-b1bf-db77851c2f71 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:30.639955 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:30.651469 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-495743e2-b660-4b91-b1bf-db77851c2f71 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:30.652911 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-495743e2-b660-4b91-b1bf-db77851c2f71 service cinder] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:30.662769 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 925/1870] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:12:30 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:12:35.322128 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a97d951-8222-4d68-88e2-2f5279a0d58f service nova] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:35.429674 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a97d951-8222-4d68-88e2-2f5279a0d58f service nova] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:35.435695 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a86ffe1a-a915-4a7d-89f8-cf10193e3f47 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:35.437033 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a86ffe1a-a915-4a7d-89f8-cf10193e3f47 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:35.437608 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a86ffe1a-a915-4a7d-89f8-cf10193e3f47 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:35.438079 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a86ffe1a-a915-4a7d-89f8-cf10193e3f47 admin admin] PATH_INFO: `/v3/users/b5a78e5138e941b586335f796244f7fc` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:35.439433 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a86ffe1a-a915-4a7d-89f8-cf10193e3f47 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b5a78e5138e941b586335f796244f7fc)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:35.442597 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a86ffe1a-a915-4a7d-89f8-cf10193e3f47 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:35.544247 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-a86ffe1a-a915-4a7d-89f8-cf10193e3f47 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b5a78e5138e941b586335f796244f7fc'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:35.557844 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-a86ffe1a-a915-4a7d-89f8-cf10193e3f47 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b5a78e5138e941b586335f796244f7fc'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:35.575993 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-a86ffe1a-a915-4a7d-89f8-cf10193e3f47 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b5a78e5138e941b586335f796244f7fc'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:35.592314 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-a86ffe1a-a915-4a7d-89f8-cf10193e3f47 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b5a78e5138e941b586335f796244f7fc'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:35.593275 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-a86ffe1a-a915-4a7d-89f8-cf10193e3f47 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b5a78e5138e941b586335f796244f7fc'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:35.605174 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 946/1871] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:12:35 2022] DELETE /identity/v3/users/b5a78e5138e941b586335f796244f7fc => generated 0 bytes in 283 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:12:35.612943 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f18e0b64-d358-41bf-8af0-39682d0fb4ff None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:35.612943 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f18e0b64-d358-41bf-8af0-39682d0fb4ff None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:35.612943 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f18e0b64-d358-41bf-8af0-39682d0fb4ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:35.722716 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-f18e0b64-d358-41bf-8af0-39682d0fb4ff None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:12:35.722716 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-f18e0b64-d358-41bf-8af0-39682d0fb4ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:35.850491 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 926/1872] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:12:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 240 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:12:35.872905 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a86ffe1a-a915-4a7d-89f8-cf10193e3f47 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:35.874793 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-a86ffe1a-a915-4a7d-89f8-cf10193e3f47 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:35.942677 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a86ffe1a-a915-4a7d-89f8-cf10193e3f47 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:35.948222 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d5d7221-528d-45a4-8186-f3ee944f6b15 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:35.948931 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d5d7221-528d-45a4-8186-f3ee944f6b15 service neutron] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:35.949741 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d5d7221-528d-45a4-8186-f3ee944f6b15 service neutron] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:35.950024 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d5d7221-528d-45a4-8186-f3ee944f6b15 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:35.959394 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:35.961964 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d5d7221-528d-45a4-8186-f3ee944f6b15 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:35.963499 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d5d7221-528d-45a4-8186-f3ee944f6b15 service neutron] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:35.981259 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 947/1873] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:12:35 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:12:36.079443 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f18e0b64-d358-41bf-8af0-39682d0fb4ff None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:36.081839 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-f18e0b64-d358-41bf-8af0-39682d0fb4ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:36.095634 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f18e0b64-d358-41bf-8af0-39682d0fb4ff None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:36.101062 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c531ac74-4841-4771-a231-237b7232591b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:36.101865 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c531ac74-4841-4771-a231-237b7232591b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:36.102143 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c531ac74-4841-4771-a231-237b7232591b admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:36.102410 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c531ac74-4841-4771-a231-237b7232591b admin admin] PATH_INFO: `/v3/users/b174111182d04974a14f44c96b178dd4` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:36.103708 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c531ac74-4841-4771-a231-237b7232591b admin admin] RBAC: Authorizing `identity:delete_user(user_id=b174111182d04974a14f44c96b178dd4)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:36.105776 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c531ac74-4841-4771-a231-237b7232591b admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:36.164450 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-c531ac74-4841-4771-a231-237b7232591b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b174111182d04974a14f44c96b178dd4'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:36.165650 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-c531ac74-4841-4771-a231-237b7232591b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b174111182d04974a14f44c96b178dd4'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:36.167366 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-c531ac74-4841-4771-a231-237b7232591b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b174111182d04974a14f44c96b178dd4'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:36.189296 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-c531ac74-4841-4771-a231-237b7232591b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b174111182d04974a14f44c96b178dd4'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:36.201588 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-c531ac74-4841-4771-a231-237b7232591b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b174111182d04974a14f44c96b178dd4'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:36.223029 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 927/1874] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:12:36 2022] DELETE /identity/v3/users/b174111182d04974a14f44c96b178dd4 => generated 0 bytes in 144 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 05:12:36.227209 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c805fa-1e91-4000-909e-df531d0f38cd None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:36.227813 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c805fa-1e91-4000-909e-df531d0f38cd None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:36.228189 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c805fa-1e91-4000-909e-df531d0f38cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:36.252856 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-58c805fa-1e91-4000-909e-df531d0f38cd None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:12:36.255001 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-58c805fa-1e91-4000-909e-df531d0f38cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:36.330541 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 948/1875] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:12:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 05:12:36.371953 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c531ac74-4841-4771-a231-237b7232591b admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:36.373677 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-c531ac74-4841-4771-a231-237b7232591b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:36.471228 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c531ac74-4841-4771-a231-237b7232591b admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:36.476578 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c11534da-c3df-4b15-b992-3db64d273a86 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:36.477304 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c11534da-c3df-4b15-b992-3db64d273a86 service neutron] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:36.477671 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c11534da-c3df-4b15-b992-3db64d273a86 service neutron] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:36.478121 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c11534da-c3df-4b15-b992-3db64d273a86 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:36.491860 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c11534da-c3df-4b15-b992-3db64d273a86 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:36.491860 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c11534da-c3df-4b15-b992-3db64d273a86 service neutron] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:36.504953 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 928/1876] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:12:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 133 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:12:36.742861 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58c805fa-1e91-4000-909e-df531d0f38cd None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:36.744438 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-58c805fa-1e91-4000-909e-df531d0f38cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:36.768426 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58c805fa-1e91-4000-909e-df531d0f38cd None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:36.779735 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7537821-455d-4a9a-bf25-00c3c00252ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:36.781480 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7537821-455d-4a9a-bf25-00c3c00252ce admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:36.782717 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7537821-455d-4a9a-bf25-00c3c00252ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:36.783198 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7537821-455d-4a9a-bf25-00c3c00252ce admin admin] PATH_INFO: `/v3/projects/bec16540a4be4e40bf3bab0ef22861b5` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:36.787625 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7537821-455d-4a9a-bf25-00c3c00252ce admin admin] RBAC: Authorizing `identity:delete_project(project_id=bec16540a4be4e40bf3bab0ef22861b5)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:36.790141 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7537821-455d-4a9a-bf25-00c3c00252ce admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:36.855102 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-d7537821-455d-4a9a-bf25-00c3c00252ce admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bec16540a4be4e40bf3bab0ef22861b5', 'request_id': 'req-d7537821-455d-4a9a-bf25-00c3c00252ce'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:36.874857 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-d7537821-455d-4a9a-bf25-00c3c00252ce admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bec16540a4be4e40bf3bab0ef22861b5', 'request_id': 'req-d7537821-455d-4a9a-bf25-00c3c00252ce'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:36.897114 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-d7537821-455d-4a9a-bf25-00c3c00252ce admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bec16540a4be4e40bf3bab0ef22861b5', 'request_id': 'req-d7537821-455d-4a9a-bf25-00c3c00252ce'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:36.914621 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-d7537821-455d-4a9a-bf25-00c3c00252ce admin admin] The token cache is being invalidate because project bec16540a4be4e40bf3bab0ef22861b5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81625) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:12:36.915226 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-d7537821-455d-4a9a-bf25-00c3c00252ce admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:36.915959 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-d7537821-455d-4a9a-bf25-00c3c00252ce admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:36.923296 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c11534da-c3df-4b15-b992-3db64d273a86 service neutron] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:36.940019 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 949/1877] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:12:36 2022] DELETE /identity/v3/projects/bec16540a4be4e40bf3bab0ef22861b5 => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:12:36.941144 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c11534da-c3df-4b15-b992-3db64d273a86 service neutron] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:36.950546 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:37.046779 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aac266ba-853b-4a61-b598-53063dad4145 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:37.047521 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aac266ba-853b-4a61-b598-53063dad4145 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:37.047874 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aac266ba-853b-4a61-b598-53063dad4145 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:37.048219 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aac266ba-853b-4a61-b598-53063dad4145 admin admin] PATH_INFO: `/v3/users/2abfa5603b39423d9c386bda4bc22603` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:37.049429 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aac266ba-853b-4a61-b598-53063dad4145 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2abfa5603b39423d9c386bda4bc22603)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:37.051910 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aac266ba-853b-4a61-b598-53063dad4145 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:37.127716 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-aac266ba-853b-4a61-b598-53063dad4145 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2abfa5603b39423d9c386bda4bc22603'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:37.128459 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-aac266ba-853b-4a61-b598-53063dad4145 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2abfa5603b39423d9c386bda4bc22603'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:37.129132 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-aac266ba-853b-4a61-b598-53063dad4145 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2abfa5603b39423d9c386bda4bc22603'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:37.149078 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-aac266ba-853b-4a61-b598-53063dad4145 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2abfa5603b39423d9c386bda4bc22603'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:37.160814 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-aac266ba-853b-4a61-b598-53063dad4145 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2abfa5603b39423d9c386bda4bc22603'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:37.180556 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 929/1878] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:12:36 2022] DELETE /identity/v3/users/2abfa5603b39423d9c386bda4bc22603 => generated 0 bytes in 258 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:12:37.339929 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7537821-455d-4a9a-bf25-00c3c00252ce admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:37.343033 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-d7537821-455d-4a9a-bf25-00c3c00252ce admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:37.366152 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7537821-455d-4a9a-bf25-00c3c00252ce admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:37.460906 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cf17016-86b6-48db-96c1-b0412bf9ade1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:37.461892 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cf17016-86b6-48db-96c1-b0412bf9ade1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:37.462318 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cf17016-86b6-48db-96c1-b0412bf9ade1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:37.462728 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cf17016-86b6-48db-96c1-b0412bf9ade1 admin admin] PATH_INFO: `/v3/projects/a7c46125ea924a3aaebdd47f16ef5de6` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:37.464158 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cf17016-86b6-48db-96c1-b0412bf9ade1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a7c46125ea924a3aaebdd47f16ef5de6)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:37.467275 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cf17016-86b6-48db-96c1-b0412bf9ade1 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:37.494034 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:37.523911 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-1cf17016-86b6-48db-96c1-b0412bf9ade1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a7c46125ea924a3aaebdd47f16ef5de6', 'request_id': 'req-1cf17016-86b6-48db-96c1-b0412bf9ade1'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:37.531795 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-1cf17016-86b6-48db-96c1-b0412bf9ade1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a7c46125ea924a3aaebdd47f16ef5de6', 'request_id': 'req-1cf17016-86b6-48db-96c1-b0412bf9ade1'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:37.538453 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-1cf17016-86b6-48db-96c1-b0412bf9ade1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a7c46125ea924a3aaebdd47f16ef5de6', 'request_id': 'req-1cf17016-86b6-48db-96c1-b0412bf9ade1'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:37.549223 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-1cf17016-86b6-48db-96c1-b0412bf9ade1 admin admin] The token cache is being invalidate because project a7c46125ea924a3aaebdd47f16ef5de6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81625) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:12:37.549517 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-1cf17016-86b6-48db-96c1-b0412bf9ade1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:37.550210 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-1cf17016-86b6-48db-96c1-b0412bf9ade1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:37.552352 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 950/1879] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:12:37 2022] DELETE /identity/v3/projects/a7c46125ea924a3aaebdd47f16ef5de6 => generated 0 bytes in 213 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:12:37.556603 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aac266ba-853b-4a61-b598-53063dad4145 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:37.558210 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-aac266ba-853b-4a61-b598-53063dad4145 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:37.572907 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aac266ba-853b-4a61-b598-53063dad4145 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:37.590490 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cf17016-86b6-48db-96c1-b0412bf9ade1 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:37.596080 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-1cf17016-86b6-48db-96c1-b0412bf9ade1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:37.640392 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dfe192f-6636-4417-92bc-13a9411e57c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:37.641014 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dfe192f-6636-4417-92bc-13a9411e57c1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:37.641289 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dfe192f-6636-4417-92bc-13a9411e57c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:37.641614 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dfe192f-6636-4417-92bc-13a9411e57c1 admin admin] PATH_INFO: `/v3/users/fb10b00acb54452abd27a8920a4965eb` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:37.642787 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dfe192f-6636-4417-92bc-13a9411e57c1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=fb10b00acb54452abd27a8920a4965eb)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:37.645377 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dfe192f-6636-4417-92bc-13a9411e57c1 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:37.665988 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cf17016-86b6-48db-96c1-b0412bf9ade1 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:37.671145 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2621f060-4117-45ae-aeef-38a2141cd5d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:37.671904 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2621f060-4117-45ae-aeef-38a2141cd5d7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:37.672254 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2621f060-4117-45ae-aeef-38a2141cd5d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:37.672588 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2621f060-4117-45ae-aeef-38a2141cd5d7 admin admin] PATH_INFO: `/v3/projects/fbc9ec8aa1f742eebc6da392cfafd613` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:37.673815 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2621f060-4117-45ae-aeef-38a2141cd5d7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fbc9ec8aa1f742eebc6da392cfafd613)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:37.676598 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2621f060-4117-45ae-aeef-38a2141cd5d7 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:37.679474 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=84,keystone:DELETE=15,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:37.699795 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-3dfe192f-6636-4417-92bc-13a9411e57c1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fb10b00acb54452abd27a8920a4965eb'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:37.700454 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-3dfe192f-6636-4417-92bc-13a9411e57c1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fb10b00acb54452abd27a8920a4965eb'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:37.701168 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-3dfe192f-6636-4417-92bc-13a9411e57c1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fb10b00acb54452abd27a8920a4965eb'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:37.713705 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-3dfe192f-6636-4417-92bc-13a9411e57c1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fb10b00acb54452abd27a8920a4965eb'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:37.721133 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-3dfe192f-6636-4417-92bc-13a9411e57c1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fb10b00acb54452abd27a8920a4965eb'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:37.727608 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-2621f060-4117-45ae-aeef-38a2141cd5d7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fbc9ec8aa1f742eebc6da392cfafd613', 'request_id': 'req-2621f060-4117-45ae-aeef-38a2141cd5d7'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:37.735606 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 930/1880] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:12:37 2022] DELETE /identity/v3/users/fb10b00acb54452abd27a8920a4965eb => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:12:37.739440 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-2621f060-4117-45ae-aeef-38a2141cd5d7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fbc9ec8aa1f742eebc6da392cfafd613', 'request_id': 'req-2621f060-4117-45ae-aeef-38a2141cd5d7'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:37.746584 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-2621f060-4117-45ae-aeef-38a2141cd5d7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fbc9ec8aa1f742eebc6da392cfafd613', 'request_id': 'req-2621f060-4117-45ae-aeef-38a2141cd5d7'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:37.759162 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-2621f060-4117-45ae-aeef-38a2141cd5d7 admin admin] The token cache is being invalidate because project fbc9ec8aa1f742eebc6da392cfafd613 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81625) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:12:37.759453 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-2621f060-4117-45ae-aeef-38a2141cd5d7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:37.760145 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-2621f060-4117-45ae-aeef-38a2141cd5d7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:37.762279 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 951/1881] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:12:37 2022] DELETE /identity/v3/projects/fbc9ec8aa1f742eebc6da392cfafd613 => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 05:12:37.766217 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dfe192f-6636-4417-92bc-13a9411e57c1 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:37.767896 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-3dfe192f-6636-4417-92bc-13a9411e57c1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:37.837752 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dfe192f-6636-4417-92bc-13a9411e57c1 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:37.842794 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bdcb8c7-6179-4d67-b696-0390cd8bb072 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:37.844184 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bdcb8c7-6179-4d67-b696-0390cd8bb072 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:37.844866 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bdcb8c7-6179-4d67-b696-0390cd8bb072 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:37.845299 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bdcb8c7-6179-4d67-b696-0390cd8bb072 admin admin] PATH_INFO: `/v3/users/c6952bfdf26440058f1391958081a8ed` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:37.847314 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bdcb8c7-6179-4d67-b696-0390cd8bb072 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c6952bfdf26440058f1391958081a8ed)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:37.849613 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bdcb8c7-6179-4d67-b696-0390cd8bb072 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:37.904616 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-2bdcb8c7-6179-4d67-b696-0390cd8bb072 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c6952bfdf26440058f1391958081a8ed'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:37.905552 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-2bdcb8c7-6179-4d67-b696-0390cd8bb072 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c6952bfdf26440058f1391958081a8ed'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:37.906355 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-2bdcb8c7-6179-4d67-b696-0390cd8bb072 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c6952bfdf26440058f1391958081a8ed'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:37.919953 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-2bdcb8c7-6179-4d67-b696-0390cd8bb072 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c6952bfdf26440058f1391958081a8ed'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:37.928163 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-2bdcb8c7-6179-4d67-b696-0390cd8bb072 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c6952bfdf26440058f1391958081a8ed'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:37.945132 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 931/1882] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:12:37 2022] DELETE /identity/v3/users/c6952bfdf26440058f1391958081a8ed => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:12:38.032749 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2621f060-4117-45ae-aeef-38a2141cd5d7 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:38.034662 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-2621f060-4117-45ae-aeef-38a2141cd5d7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:38.101775 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:DELETE=14,keystone:SELECT=84,keystone:INSERT=2 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:38.136592 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2621f060-4117-45ae-aeef-38a2141cd5d7 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:38.212305 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e62f55c-3406-480b-a2b1-af16519cddde admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:38.212942 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e62f55c-3406-480b-a2b1-af16519cddde admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:38.213217 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e62f55c-3406-480b-a2b1-af16519cddde admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:38.213487 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e62f55c-3406-480b-a2b1-af16519cddde admin admin] PATH_INFO: `/v3/projects/e972bb0e3d8f446caf920c6f378df9bf` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:38.214612 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e62f55c-3406-480b-a2b1-af16519cddde admin admin] RBAC: Authorizing `identity:delete_project(project_id=e972bb0e3d8f446caf920c6f378df9bf)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:38.216619 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e62f55c-3406-480b-a2b1-af16519cddde admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:38.274664 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-7e62f55c-3406-480b-a2b1-af16519cddde admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e972bb0e3d8f446caf920c6f378df9bf', 'request_id': 'req-7e62f55c-3406-480b-a2b1-af16519cddde'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:38.287981 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-7e62f55c-3406-480b-a2b1-af16519cddde admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e972bb0e3d8f446caf920c6f378df9bf', 'request_id': 'req-7e62f55c-3406-480b-a2b1-af16519cddde'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:38.306223 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-7e62f55c-3406-480b-a2b1-af16519cddde admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e972bb0e3d8f446caf920c6f378df9bf', 'request_id': 'req-7e62f55c-3406-480b-a2b1-af16519cddde'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:38.319069 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-7e62f55c-3406-480b-a2b1-af16519cddde admin admin] The token cache is being invalidate because project e972bb0e3d8f446caf920c6f378df9bf was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81625) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:12:38.319362 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-7e62f55c-3406-480b-a2b1-af16519cddde admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:38.321133 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-7e62f55c-3406-480b-a2b1-af16519cddde admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:38.323224 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 952/1883] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:12:38 2022] DELETE /identity/v3/projects/e972bb0e3d8f446caf920c6f378df9bf => generated 0 bytes in 291 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:12:38.323824 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bdcb8c7-6179-4d67-b696-0390cd8bb072 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:38.325296 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-2bdcb8c7-6179-4d67-b696-0390cd8bb072 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:38.326894 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e62f55c-3406-480b-a2b1-af16519cddde admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:38.328261 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-7e62f55c-3406-480b-a2b1-af16519cddde admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:38.339504 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bdcb8c7-6179-4d67-b696-0390cd8bb072 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:38.373928 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=83,keystone:DELETE=15,keystone:INSERT=2 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:38.420504 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e62f55c-3406-480b-a2b1-af16519cddde admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:38.426951 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08e10e93-5f8c-4765-b4d2-da353c1fdcbe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:38.427595 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e10e93-5f8c-4765-b4d2-da353c1fdcbe admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:38.427874 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e10e93-5f8c-4765-b4d2-da353c1fdcbe admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:38.428262 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e10e93-5f8c-4765-b4d2-da353c1fdcbe admin admin] PATH_INFO: `/v3/users/712f11bcd1644a69858ca0e891faeb3c` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:38.429603 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08e10e93-5f8c-4765-b4d2-da353c1fdcbe admin admin] RBAC: Authorizing `identity:delete_user(user_id=712f11bcd1644a69858ca0e891faeb3c)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:38.431410 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08e10e93-5f8c-4765-b4d2-da353c1fdcbe admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:38.434596 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cabb289f-6cce-424f-b534-27e5a6702a55 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:38.435270 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cabb289f-6cce-424f-b534-27e5a6702a55 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:38.435678 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cabb289f-6cce-424f-b534-27e5a6702a55 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:38.435960 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cabb289f-6cce-424f-b534-27e5a6702a55 admin admin] PATH_INFO: `/v3/projects/1d204a50ea3440de85d9e216ca404ac9` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:38.437103 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cabb289f-6cce-424f-b534-27e5a6702a55 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1d204a50ea3440de85d9e216ca404ac9)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:38.439076 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cabb289f-6cce-424f-b534-27e5a6702a55 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:38.467572 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:38.513323 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-08e10e93-5f8c-4765-b4d2-da353c1fdcbe admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '712f11bcd1644a69858ca0e891faeb3c'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:38.516942 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-cabb289f-6cce-424f-b534-27e5a6702a55 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1d204a50ea3440de85d9e216ca404ac9', 'request_id': 'req-cabb289f-6cce-424f-b534-27e5a6702a55'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:38.529077 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-08e10e93-5f8c-4765-b4d2-da353c1fdcbe admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '712f11bcd1644a69858ca0e891faeb3c'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:38.544624 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-08e10e93-5f8c-4765-b4d2-da353c1fdcbe admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '712f11bcd1644a69858ca0e891faeb3c'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:38.547564 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-cabb289f-6cce-424f-b534-27e5a6702a55 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1d204a50ea3440de85d9e216ca404ac9', 'request_id': 'req-cabb289f-6cce-424f-b534-27e5a6702a55'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:38.554699 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-08e10e93-5f8c-4765-b4d2-da353c1fdcbe admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '712f11bcd1644a69858ca0e891faeb3c'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:38.555830 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-08e10e93-5f8c-4765-b4d2-da353c1fdcbe admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '712f11bcd1644a69858ca0e891faeb3c'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:38.557790 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-cabb289f-6cce-424f-b534-27e5a6702a55 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1d204a50ea3440de85d9e216ca404ac9', 'request_id': 'req-cabb289f-6cce-424f-b534-27e5a6702a55'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:38.562341 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 953/1884] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:12:38 2022] DELETE /identity/v3/users/712f11bcd1644a69858ca0e891faeb3c => generated 0 bytes in 236 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 05:12:38.581007 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-cabb289f-6cce-424f-b534-27e5a6702a55 admin admin] The token cache is being invalidate because project 1d204a50ea3440de85d9e216ca404ac9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81624) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:12:38.581213 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-cabb289f-6cce-424f-b534-27e5a6702a55 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:38.581885 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-cabb289f-6cce-424f-b534-27e5a6702a55 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:38.583426 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 932/1885] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:12:38 2022] DELETE /identity/v3/projects/1d204a50ea3440de85d9e216ca404ac9 => generated 0 bytes in 260 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:12:38.593871 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-611992c6-a3e6-4e6d-9ae3-e4c0b2b9b36f None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:38.597938 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-611992c6-a3e6-4e6d-9ae3-e4c0b2b9b36f None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:38.598236 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-611992c6-a3e6-4e6d-9ae3-e4c0b2b9b36f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:38.628788 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-611992c6-a3e6-4e6d-9ae3-e4c0b2b9b36f None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:12:38.630304 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-611992c6-a3e6-4e6d-9ae3-e4c0b2b9b36f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:38.720804 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 954/1886] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:12:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:12:38.729894 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfce1b66-3bae-43f1-8b7d-f11fcd0c24cc None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:38.730670 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfce1b66-3bae-43f1-8b7d-f11fcd0c24cc None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:38.730670 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfce1b66-3bae-43f1-8b7d-f11fcd0c24cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:38.755261 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-dfce1b66-3bae-43f1-8b7d-f11fcd0c24cc None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:12:38.759813 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-dfce1b66-3bae-43f1-8b7d-f11fcd0c24cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:38.775202 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 933/1887] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:12:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 47 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:12:38.776468 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-611992c6-a3e6-4e6d-9ae3-e4c0b2b9b36f None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:38.793601 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-611992c6-a3e6-4e6d-9ae3-e4c0b2b9b36f None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:38.801449 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37bd0808-bcb2-4146-9a77-29ba53a7321e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:38.802199 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37bd0808-bcb2-4146-9a77-29ba53a7321e admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:38.802550 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37bd0808-bcb2-4146-9a77-29ba53a7321e admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:38.803448 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37bd0808-bcb2-4146-9a77-29ba53a7321e admin admin] PATH_INFO: `/v3/domains` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:38.804355 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37bd0808-bcb2-4146-9a77-29ba53a7321e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:12:38.804736 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37bd0808-bcb2-4146-9a77-29ba53a7321e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:38.806425 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37bd0808-bcb2-4146-9a77-29ba53a7321e admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:38.827636 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 955/1888] 10.222.0.77 () {58 vars in 1153 bytes} [Tue Aug 30 05:12:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 05:12:38.827991 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfce1b66-3bae-43f1-8b7d-f11fcd0c24cc None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:38.838712 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfce1b66-3bae-43f1-8b7d-f11fcd0c24cc None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:38.844318 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d1fee5a-c1d7-4776-8ff4-9389a168205f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:38.845372 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d1fee5a-c1d7-4776-8ff4-9389a168205f admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:38.845719 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d1fee5a-c1d7-4776-8ff4-9389a168205f admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:38.846066 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d1fee5a-c1d7-4776-8ff4-9389a168205f admin admin] PATH_INFO: `/v3/projects` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:38.847477 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d1fee5a-c1d7-4776-8ff4-9389a168205f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:38.849752 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d1fee5a-c1d7-4776-8ff4-9389a168205f admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:38.894671 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 934/1889] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:12:38 2022] POST /identity/v3/projects => generated 383 bytes in 68 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:12:38.899933 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37bd0808-bcb2-4146-9a77-29ba53a7321e admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:38.912239 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37bd0808-bcb2-4146-9a77-29ba53a7321e admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:38.979668 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86a3d0ae-67f2-45aa-b30b-c3e3ca274c09 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:38.980622 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a3d0ae-67f2-45aa-b30b-c3e3ca274c09 admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:38.980931 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a3d0ae-67f2-45aa-b30b-c3e3ca274c09 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:38.981347 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a3d0ae-67f2-45aa-b30b-c3e3ca274c09 admin admin] PATH_INFO: `/v3/users` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:38.981925 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86a3d0ae-67f2-45aa-b30b-c3e3ca274c09 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:38.984217 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86a3d0ae-67f2-45aa-b30b-c3e3ca274c09 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:39.014052 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 956/1890] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:12:38 2022] POST /identity/v3/users => generated 292 bytes in 114 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:12:39.018474 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d1fee5a-c1d7-4776-8ff4-9389a168205f admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:39.038368 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d1fee5a-c1d7-4776-8ff4-9389a168205f admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:39.054980 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45374d4f-183e-4128-a5c9-45e6d8219f8b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:39.055698 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45374d4f-183e-4128-a5c9-45e6d8219f8b admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:39.056054 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45374d4f-183e-4128-a5c9-45e6d8219f8b admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:39.056569 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45374d4f-183e-4128-a5c9-45e6d8219f8b admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:39.057298 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45374d4f-183e-4128-a5c9-45e6d8219f8b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:39.059044 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45374d4f-183e-4128-a5c9-45e6d8219f8b admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:39.077078 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 935/1891] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:12:39 2022] GET /identity/v3/roles => generated 1420 bytes in 60 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:12:39.082134 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86a3d0ae-67f2-45aa-b30b-c3e3ca274c09 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:39.093345 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86a3d0ae-67f2-45aa-b30b-c3e3ca274c09 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:39.099865 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b3b6955-3ecf-4a54-837b-bc2df0bf7a0c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:39.100691 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b3b6955-3ecf-4a54-837b-bc2df0bf7a0c admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:39.100838 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b3b6955-3ecf-4a54-837b-bc2df0bf7a0c admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:39.101025 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b3b6955-3ecf-4a54-837b-bc2df0bf7a0c admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:39.101677 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b3b6955-3ecf-4a54-837b-bc2df0bf7a0c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:39.103370 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b3b6955-3ecf-4a54-837b-bc2df0bf7a0c admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:39.119233 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 957/1892] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:12:39 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 05:12:39.120497 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45374d4f-183e-4128-a5c9-45e6d8219f8b admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:39.131515 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45374d4f-183e-4128-a5c9-45e6d8219f8b admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:39.136643 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70e62812-f3fd-4d23-91cc-83fe970e9808 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:39.137418 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70e62812-f3fd-4d23-91cc-83fe970e9808 admin admin] REQUEST_METHOD: `PUT` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:39.137773 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70e62812-f3fd-4d23-91cc-83fe970e9808 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:39.138119 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70e62812-f3fd-4d23-91cc-83fe970e9808 admin admin] PATH_INFO: `/v3/projects/f84f3c42dcc54722b46789217a4efd66/users/9372a0808794476885c8e73c97387970/roles/6f03d86980c34c36b44a8d6764ff4fda` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:39.156363 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70e62812-f3fd-4d23-91cc-83fe970e9808 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f84f3c42dcc54722b46789217a4efd66, user_id=9372a0808794476885c8e73c97387970, role_id=6f03d86980c34c36b44a8d6764ff4fda)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:39.159944 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70e62812-f3fd-4d23-91cc-83fe970e9808 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:39.175947 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 936/1893] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:12:39 2022] PUT /identity/v3/projects/f84f3c42dcc54722b46789217a4efd66/users/9372a0808794476885c8e73c97387970/roles/6f03d86980c34c36b44a8d6764ff4fda => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:12:39.192057 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d8b4ad0-fc9b-4ced-9cb1-2562a248ec31 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:39.192582 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d8b4ad0-fc9b-4ced-9cb1-2562a248ec31 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:39.192959 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d8b4ad0-fc9b-4ced-9cb1-2562a248ec31 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:39.204308 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:DELETE=8,keystone:SELECT=88,keystone:INSERT=4 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:39.241233 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-5d8b4ad0-fc9b-4ced-9cb1-2562a248ec31 None None] MFA Rules not processed for user `9372a0808794476885c8e73c97387970`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:12:39.242583 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-5d8b4ad0-fc9b-4ced-9cb1-2562a248ec31 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:39.337536 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 958/1894] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:12:39 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:12:39.337812 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70e62812-f3fd-4d23-91cc-83fe970e9808 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:39.454370 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:39.470521 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70e62812-f3fd-4d23-91cc-83fe970e9808 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:39.479297 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-534c49e9-0ee1-4722-8de1-b6b8a24959cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:39.482942 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-534c49e9-0ee1-4722-8de1-b6b8a24959cf admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:39.483525 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-534c49e9-0ee1-4722-8de1-b6b8a24959cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:39.483998 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-534c49e9-0ee1-4722-8de1-b6b8a24959cf admin admin] PATH_INFO: `/v3/projects` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:39.485480 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-534c49e9-0ee1-4722-8de1-b6b8a24959cf admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:39.488334 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-534c49e9-0ee1-4722-8de1-b6b8a24959cf admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:39.528776 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 937/1895] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:12:39 2022] POST /identity/v3/projects => generated 383 bytes in 193 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:12:39.530116 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d8b4ad0-fc9b-4ced-9cb1-2562a248ec31 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:39.545279 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d8b4ad0-fc9b-4ced-9cb1-2562a248ec31 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:39.624005 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f28e91a-31ed-4198-9ad8-a8ecc106ceb9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:39.624629 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f28e91a-31ed-4198-9ad8-a8ecc106ceb9 admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:39.624910 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f28e91a-31ed-4198-9ad8-a8ecc106ceb9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:39.625183 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f28e91a-31ed-4198-9ad8-a8ecc106ceb9 admin admin] PATH_INFO: `/v3/users` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:39.626344 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f28e91a-31ed-4198-9ad8-a8ecc106ceb9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:39.629322 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f28e91a-31ed-4198-9ad8-a8ecc106ceb9 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:39.655986 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 959/1896] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:12:39 2022] POST /identity/v3/users => generated 298 bytes in 126 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:12:39.659206 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-534c49e9-0ee1-4722-8de1-b6b8a24959cf admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:39.669519 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-534c49e9-0ee1-4722-8de1-b6b8a24959cf admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:39.674246 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fec1c836-f383-4a13-afe7-768c7368ad16 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:39.674905 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec1c836-f383-4a13-afe7-768c7368ad16 admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:39.675188 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec1c836-f383-4a13-afe7-768c7368ad16 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:39.675460 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec1c836-f383-4a13-afe7-768c7368ad16 admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:39.676156 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fec1c836-f383-4a13-afe7-768c7368ad16 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:39.677679 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fec1c836-f383-4a13-afe7-768c7368ad16 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:39.689712 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 938/1897] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:12:39 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:12:39.703130 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f28e91a-31ed-4198-9ad8-a8ecc106ceb9 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:39.714154 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f28e91a-31ed-4198-9ad8-a8ecc106ceb9 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:39.720660 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc1b125c-6cbf-4cf0-90ac-a1bca20a7557 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:39.721406 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc1b125c-6cbf-4cf0-90ac-a1bca20a7557 admin admin] REQUEST_METHOD: `PUT` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:39.721641 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc1b125c-6cbf-4cf0-90ac-a1bca20a7557 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:39.721900 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc1b125c-6cbf-4cf0-90ac-a1bca20a7557 admin admin] PATH_INFO: `/v3/projects/a626219ad6c848509fe576bd569ae7c6/users/ce7cb64caf0843bbad8b508f4b2acd9d/roles/9d9d5d2c270146488249b7ed943280db` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:39.740879 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc1b125c-6cbf-4cf0-90ac-a1bca20a7557 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a626219ad6c848509fe576bd569ae7c6, user_id=ce7cb64caf0843bbad8b508f4b2acd9d, role_id=9d9d5d2c270146488249b7ed943280db)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:39.744710 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc1b125c-6cbf-4cf0-90ac-a1bca20a7557 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:39.761070 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 960/1898] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:12:39 2022] PUT /identity/v3/projects/a626219ad6c848509fe576bd569ae7c6/users/ce7cb64caf0843bbad8b508f4b2acd9d/roles/9d9d5d2c270146488249b7ed943280db => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:12:39.764966 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fec1c836-f383-4a13-afe7-768c7368ad16 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:39.776402 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fec1c836-f383-4a13-afe7-768c7368ad16 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:39.826165 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc1b125c-6cbf-4cf0-90ac-a1bca20a7557 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:39.828097 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-fc1b125c-6cbf-4cf0-90ac-a1bca20a7557 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:39.840535 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a055997-27e6-4184-b342-230e5bcbced0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:39.841376 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a055997-27e6-4184-b342-230e5bcbced0 admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:39.841663 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a055997-27e6-4184-b342-230e5bcbced0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:39.842032 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a055997-27e6-4184-b342-230e5bcbced0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:39.842999 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a055997-27e6-4184-b342-230e5bcbced0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:39.845274 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a055997-27e6-4184-b342-230e5bcbced0 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:39.862068 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 939/1899] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:12:39 2022] GET /identity/v3/roles => generated 1420 bytes in 94 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:12:39.863632 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a055997-27e6-4184-b342-230e5bcbced0 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:39.874027 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a055997-27e6-4184-b342-230e5bcbced0 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:39.879611 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d912cef-6a56-4d58-b262-9a22566fa503 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:39.880260 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d912cef-6a56-4d58-b262-9a22566fa503 admin admin] REQUEST_METHOD: `PUT` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:39.880538 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d912cef-6a56-4d58-b262-9a22566fa503 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:39.880810 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d912cef-6a56-4d58-b262-9a22566fa503 admin admin] PATH_INFO: `/v3/projects/a626219ad6c848509fe576bd569ae7c6/users/ce7cb64caf0843bbad8b508f4b2acd9d/roles/9d9d5d2c270146488249b7ed943280db` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:39.882767 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d912cef-6a56-4d58-b262-9a22566fa503 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a626219ad6c848509fe576bd569ae7c6, user_id=ce7cb64caf0843bbad8b508f4b2acd9d, role_id=9d9d5d2c270146488249b7ed943280db)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:39.885702 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d912cef-6a56-4d58-b262-9a22566fa503 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:39.896004 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:39.897135 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 940/1900] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:12:39 2022] PUT /identity/v3/projects/a626219ad6c848509fe576bd569ae7c6/users/ce7cb64caf0843bbad8b508f4b2acd9d/roles/9d9d5d2c270146488249b7ed943280db => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 05:12:39.907861 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc1b125c-6cbf-4cf0-90ac-a1bca20a7557 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:39.910273 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b9ab795-e738-4a5e-b95d-c540078e065a None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:39.910565 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b9ab795-e738-4a5e-b95d-c540078e065a None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:39.910854 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b9ab795-e738-4a5e-b95d-c540078e065a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:39.948868 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-0b9ab795-e738-4a5e-b95d-c540078e065a None None] MFA Rules not processed for user `ce7cb64caf0843bbad8b508f4b2acd9d`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:12:39.950144 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-0b9ab795-e738-4a5e-b95d-c540078e065a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:39.979536 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9ff7f7d-669b-4274-a811-fe1dd590b9e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:39.980170 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9ff7f7d-669b-4274-a811-fe1dd590b9e4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:39.980450 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9ff7f7d-669b-4274-a811-fe1dd590b9e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:39.980745 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9ff7f7d-669b-4274-a811-fe1dd590b9e4 admin admin] PATH_INFO: `/v3/projects/6fb323c928e042eb94b090c8c875af91` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:39.981870 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9ff7f7d-669b-4274-a811-fe1dd590b9e4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6fb323c928e042eb94b090c8c875af91)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:39.983824 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9ff7f7d-669b-4274-a811-fe1dd590b9e4 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:40.028701 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 941/1901] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:12:39 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:12:40.031917 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-d9ff7f7d-669b-4274-a811-fe1dd590b9e4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6fb323c928e042eb94b090c8c875af91', 'request_id': 'req-d9ff7f7d-669b-4274-a811-fe1dd590b9e4'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:40.040453 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-d9ff7f7d-669b-4274-a811-fe1dd590b9e4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6fb323c928e042eb94b090c8c875af91', 'request_id': 'req-d9ff7f7d-669b-4274-a811-fe1dd590b9e4'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:40.041023 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b9ab795-e738-4a5e-b95d-c540078e065a None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:40.042960 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-0b9ab795-e738-4a5e-b95d-c540078e065a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:40.047099 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-d9ff7f7d-669b-4274-a811-fe1dd590b9e4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6fb323c928e042eb94b090c8c875af91', 'request_id': 'req-d9ff7f7d-669b-4274-a811-fe1dd590b9e4'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:40.058499 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-d9ff7f7d-669b-4274-a811-fe1dd590b9e4 admin admin] The token cache is being invalidate because project 6fb323c928e042eb94b090c8c875af91 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81625) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:12:40.058769 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-d9ff7f7d-669b-4274-a811-fe1dd590b9e4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:40.059450 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-d9ff7f7d-669b-4274-a811-fe1dd590b9e4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:40.061024 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 961/1902] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:12:39 2022] DELETE /identity/v3/projects/6fb323c928e042eb94b090c8c875af91 => generated 0 bytes in 235 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 05:12:40.066974 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50ccd920-b24a-497f-922f-86f79ab3bf81 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:40.067258 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50ccd920-b24a-497f-922f-86f79ab3bf81 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:40.067526 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50ccd920-b24a-497f-922f-86f79ab3bf81 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:40.078555 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:40.087788 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-50ccd920-b24a-497f-922f-86f79ab3bf81 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:12:40.088925 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-50ccd920-b24a-497f-922f-86f79ab3bf81 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:40.103212 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b9ab795-e738-4a5e-b95d-c540078e065a None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:40.148791 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 962/1903] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:12:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 05:12:40.151553 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59470b17-4891-46eb-a634-c71e3e46f30f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '019a79b2affe478b8a30d1c538d0415c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:40.152199 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59470b17-4891-46eb-a634-c71e3e46f30f service cinder] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:40.152476 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59470b17-4891-46eb-a634-c71e3e46f30f service cinder] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:40.152739 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59470b17-4891-46eb-a634-c71e3e46f30f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:40.154432 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-59470b17-4891-46eb-a634-c71e3e46f30f service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:40.156536 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-187f0f15-ad20-46dc-9a4f-5edc19b12e95 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:40.156824 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-187f0f15-ad20-46dc-9a4f-5edc19b12e95 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:40.157097 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-187f0f15-ad20-46dc-9a4f-5edc19b12e95 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:40.176448 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-187f0f15-ad20-46dc-9a4f-5edc19b12e95 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:12:40.177644 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-187f0f15-ad20-46dc-9a4f-5edc19b12e95 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:40.185506 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 963/1904] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:12:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 05:12:40.189807 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-187f0f15-ad20-46dc-9a4f-5edc19b12e95 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:40.200299 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-187f0f15-ad20-46dc-9a4f-5edc19b12e95 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:40.205149 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a568e6a4-065f-421d-a870-88e4ddf179c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:40.205733 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a568e6a4-065f-421d-a870-88e4ddf179c5 admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:40.206012 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a568e6a4-065f-421d-a870-88e4ddf179c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:40.206283 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a568e6a4-065f-421d-a870-88e4ddf179c5 admin admin] PATH_INFO: `/v3/domains` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:40.206953 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a568e6a4-065f-421d-a870-88e4ddf179c5 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:12:40.207254 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a568e6a4-065f-421d-a870-88e4ddf179c5 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:40.208838 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a568e6a4-065f-421d-a870-88e4ddf179c5 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:40.213821 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59470b17-4891-46eb-a634-c71e3e46f30f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:40.215270 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59470b17-4891-46eb-a634-c71e3e46f30f service cinder] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:40.225279 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 964/1905] 10.222.0.77 () {58 vars in 1153 bytes} [Tue Aug 30 05:12:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 05:12:40.225473 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 942/1906] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:12:40 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 185 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 05:12:40.226482 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a568e6a4-065f-421d-a870-88e4ddf179c5 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:40.236297 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a568e6a4-065f-421d-a870-88e4ddf179c5 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:40.241477 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9587d3a-fd50-4b61-83c6-6be1ef15b8eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:40.242126 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9587d3a-fd50-4b61-83c6-6be1ef15b8eb admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:40.242473 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9587d3a-fd50-4b61-83c6-6be1ef15b8eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:40.242810 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9587d3a-fd50-4b61-83c6-6be1ef15b8eb admin admin] PATH_INFO: `/v3/projects` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:40.243539 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9587d3a-fd50-4b61-83c6-6be1ef15b8eb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:40.245474 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9587d3a-fd50-4b61-83c6-6be1ef15b8eb admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:40.260085 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:40.275197 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 965/1907] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:12:40 2022] POST /identity/v3/projects => generated 365 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 05:12:40.279407 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59470b17-4891-46eb-a634-c71e3e46f30f service cinder] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:40.290858 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59470b17-4891-46eb-a634-c71e3e46f30f service cinder] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:40.348994 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4077b58a-fd70-40ad-a20b-7f2e5ca5f6ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:40.349549 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4077b58a-fd70-40ad-a20b-7f2e5ca5f6ab admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:40.349824 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4077b58a-fd70-40ad-a20b-7f2e5ca5f6ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:40.350091 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4077b58a-fd70-40ad-a20b-7f2e5ca5f6ab admin admin] PATH_INFO: `/v3/users` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:40.350707 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4077b58a-fd70-40ad-a20b-7f2e5ca5f6ab admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:40.352727 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4077b58a-fd70-40ad-a20b-7f2e5ca5f6ab admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:40.377591 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 943/1908] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:12:40 2022] POST /identity/v3/users => generated 283 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 05:12:40.379084 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9587d3a-fd50-4b61-83c6-6be1ef15b8eb admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:40.389228 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9587d3a-fd50-4b61-83c6-6be1ef15b8eb admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:40.393845 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38419f42-1604-4ebf-852d-076a603da8d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:40.394441 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38419f42-1604-4ebf-852d-076a603da8d9 admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:40.394736 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38419f42-1604-4ebf-852d-076a603da8d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:40.395022 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38419f42-1604-4ebf-852d-076a603da8d9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:40.395659 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38419f42-1604-4ebf-852d-076a603da8d9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:40.397240 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38419f42-1604-4ebf-852d-076a603da8d9 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:40.407785 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 966/1909] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:12:40 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:12:40.412217 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4077b58a-fd70-40ad-a20b-7f2e5ca5f6ab admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:40.423658 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4077b58a-fd70-40ad-a20b-7f2e5ca5f6ab admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:40.432398 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c86f945d-ed21-4959-860d-ffec05c364a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:40.433103 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c86f945d-ed21-4959-860d-ffec05c364a0 admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:40.433442 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c86f945d-ed21-4959-860d-ffec05c364a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:40.433774 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c86f945d-ed21-4959-860d-ffec05c364a0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:40.435135 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c86f945d-ed21-4959-860d-ffec05c364a0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:40.438176 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c86f945d-ed21-4959-860d-ffec05c364a0 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:40.455566 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 944/1910] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:12:40 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:12:40.458267 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38419f42-1604-4ebf-852d-076a603da8d9 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:40.486858 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38419f42-1604-4ebf-852d-076a603da8d9 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:40.497919 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51776bf6-4eb5-4693-bbf2-ccdfe5291b53 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:40.499182 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51776bf6-4eb5-4693-bbf2-ccdfe5291b53 admin admin] REQUEST_METHOD: `PUT` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:40.499612 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51776bf6-4eb5-4693-bbf2-ccdfe5291b53 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:40.500117 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51776bf6-4eb5-4693-bbf2-ccdfe5291b53 admin admin] PATH_INFO: `/v3/projects/2b930d0a13b84f7baa5012fc1d686b4c/users/05cb928547c94c74a7e89c39b0cc1fcf/roles/6f03d86980c34c36b44a8d6764ff4fda` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:40.526927 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51776bf6-4eb5-4693-bbf2-ccdfe5291b53 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2b930d0a13b84f7baa5012fc1d686b4c, user_id=05cb928547c94c74a7e89c39b0cc1fcf, role_id=6f03d86980c34c36b44a8d6764ff4fda)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:40.531323 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51776bf6-4eb5-4693-bbf2-ccdfe5291b53 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:40.551112 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 967/1911] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:12:40 2022] PUT /identity/v3/projects/2b930d0a13b84f7baa5012fc1d686b4c/users/05cb928547c94c74a7e89c39b0cc1fcf/roles/6f03d86980c34c36b44a8d6764ff4fda => generated 0 bytes in 95 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 05:12:40.563365 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87c91de2-9b81-4712-a190-b227e92b8d83 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:40.563695 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87c91de2-9b81-4712-a190-b227e92b8d83 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:40.564017 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87c91de2-9b81-4712-a190-b227e92b8d83 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:40.631130 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-87c91de2-9b81-4712-a190-b227e92b8d83 None None] MFA Rules not processed for user `05cb928547c94c74a7e89c39b0cc1fcf`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:12:40.636232 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-87c91de2-9b81-4712-a190-b227e92b8d83 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:40.652103 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:40.728588 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 945/1912] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:12:40 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:12:40.739923 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51776bf6-4eb5-4693-bbf2-ccdfe5291b53 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:40.741619 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-51776bf6-4eb5-4693-bbf2-ccdfe5291b53 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:40.799531 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51776bf6-4eb5-4693-bbf2-ccdfe5291b53 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:40.803189 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-517c0cc8-46e8-4d04-85d6-f6eeb1989dda service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '019a79b2affe478b8a30d1c538d0415c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:40.803790 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-517c0cc8-46e8-4d04-85d6-f6eeb1989dda service cinder] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:40.804133 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-517c0cc8-46e8-4d04-85d6-f6eeb1989dda service cinder] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:40.804409 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-517c0cc8-46e8-4d04-85d6-f6eeb1989dda service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:40.813409 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-517c0cc8-46e8-4d04-85d6-f6eeb1989dda service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:40.814938 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-517c0cc8-46e8-4d04-85d6-f6eeb1989dda service cinder] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:40.829774 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 968/1913] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:12:40 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:12:45.078411 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16e5cc80-7be4-4e99-bb7f-8b160de08455 None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:45.078411 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16e5cc80-7be4-4e99-bb7f-8b160de08455 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:45.078411 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16e5cc80-7be4-4e99-bb7f-8b160de08455 None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:45.078411 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 946/1914] 10.222.0.77 () {54 vars in 921 bytes} [Tue Aug 30 05:12:45 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:12:45.093384 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-645d5fed-620e-4631-9cae-c2b8127f397e None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:45.093895 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-645d5fed-620e-4631-9cae-c2b8127f397e None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:45.093993 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-645d5fed-620e-4631-9cae-c2b8127f397e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:45.135627 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-645d5fed-620e-4631-9cae-c2b8127f397e None None] MFA Rules not processed for user `886eb5e00bbf4ec292eb962b665b37d0`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:12:45.136863 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-645d5fed-620e-4631-9cae-c2b8127f397e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:45.249302 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 969/1915] 10.222.0.77 () {58 vars in 1018 bytes} [Tue Aug 30 05:12:45 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:12:45.277764 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16e5cc80-7be4-4e99-bb7f-8b160de08455 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:45.279755 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-16e5cc80-7be4-4e99-bb7f-8b160de08455 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:45.306083 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16e5cc80-7be4-4e99-bb7f-8b160de08455 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:45.312936 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06272d3c-cfb5-4b5c-9507-a0ccad22722c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '886eb5e00bbf4ec292eb962b665b37d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:45.313562 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06272d3c-cfb5-4b5c-9507-a0ccad22722c service nova] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:45.313853 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06272d3c-cfb5-4b5c-9507-a0ccad22722c service nova] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:45.314109 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06272d3c-cfb5-4b5c-9507-a0ccad22722c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:45.324963 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06272d3c-cfb5-4b5c-9507-a0ccad22722c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:45.326646 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06272d3c-cfb5-4b5c-9507-a0ccad22722c service nova] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:45.339876 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 947/1916] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:12:45 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 63 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 05:12:45.492531 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-645d5fed-620e-4631-9cae-c2b8127f397e None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:45.493916 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-645d5fed-620e-4631-9cae-c2b8127f397e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:45.547434 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:INSERT=2,keystone:SELECT=98 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:45.564578 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-645d5fed-620e-4631-9cae-c2b8127f397e None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:45.569977 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c18b8eaa-f132-4567-89c3-80ed8ae4141a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:45.570790 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c18b8eaa-f132-4567-89c3-80ed8ae4141a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:45.571108 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c18b8eaa-f132-4567-89c3-80ed8ae4141a admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:45.571383 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c18b8eaa-f132-4567-89c3-80ed8ae4141a admin admin] PATH_INFO: `/v3/users/e904acf61724406381ad67dbb2f3ad06` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:45.572510 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c18b8eaa-f132-4567-89c3-80ed8ae4141a admin admin] RBAC: Authorizing `identity:delete_user(user_id=e904acf61724406381ad67dbb2f3ad06)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:45.574473 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c18b8eaa-f132-4567-89c3-80ed8ae4141a admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:45.643485 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-c18b8eaa-f132-4567-89c3-80ed8ae4141a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e904acf61724406381ad67dbb2f3ad06'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:45.652017 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-c18b8eaa-f132-4567-89c3-80ed8ae4141a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e904acf61724406381ad67dbb2f3ad06'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:45.684684 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-c18b8eaa-f132-4567-89c3-80ed8ae4141a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e904acf61724406381ad67dbb2f3ad06'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:45.699936 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-c18b8eaa-f132-4567-89c3-80ed8ae4141a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e904acf61724406381ad67dbb2f3ad06'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:45.702322 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-c18b8eaa-f132-4567-89c3-80ed8ae4141a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e904acf61724406381ad67dbb2f3ad06'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:45.710585 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 970/1917] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:12:45 2022] DELETE /identity/v3/users/e904acf61724406381ad67dbb2f3ad06 => generated 0 bytes in 218 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:12:45.710863 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06272d3c-cfb5-4b5c-9507-a0ccad22722c service nova] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:45.714844 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe596629-e928-40e8-a9ed-e6764408f711 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:45.715327 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe596629-e928-40e8-a9ed-e6764408f711 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:45.715780 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe596629-e928-40e8-a9ed-e6764408f711 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:45.723729 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06272d3c-cfb5-4b5c-9507-a0ccad22722c service nova] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:45.740496 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-fe596629-e928-40e8-a9ed-e6764408f711 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:12:45.741837 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-fe596629-e928-40e8-a9ed-e6764408f711 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:45.836661 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa1546a4-6408-48b1-99b6-eb424e587575 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '019a79b2affe478b8a30d1c538d0415c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:45.837632 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa1546a4-6408-48b1-99b6-eb424e587575 service cinder] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:45.837912 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa1546a4-6408-48b1-99b6-eb424e587575 service cinder] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:45.838181 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa1546a4-6408-48b1-99b6-eb424e587575 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:45.840197 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-fa1546a4-6408-48b1-99b6-eb424e587575 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:45.849235 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 971/1918] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:12:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 05:12:45.864376 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe596629-e928-40e8-a9ed-e6764408f711 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:45.871746 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-fe596629-e928-40e8-a9ed-e6764408f711 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:45.941148 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa1546a4-6408-48b1-99b6-eb424e587575 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:45.942591 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa1546a4-6408-48b1-99b6-eb424e587575 service cinder] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:45.955096 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 948/1919] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:12:45 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 245 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:12:45.993693 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe596629-e928-40e8-a9ed-e6764408f711 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:45.998601 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-792dd9a4-9d90-4e3f-a5da-7e035f20c9c7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:45.999317 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-792dd9a4-9d90-4e3f-a5da-7e035f20c9c7 service neutron] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:45.999681 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-792dd9a4-9d90-4e3f-a5da-7e035f20c9c7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:45.999965 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-792dd9a4-9d90-4e3f-a5da-7e035f20c9c7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:46.014411 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-792dd9a4-9d90-4e3f-a5da-7e035f20c9c7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:46.016271 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-792dd9a4-9d90-4e3f-a5da-7e035f20c9c7 service neutron] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:46.027853 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 972/1920] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:12:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 05:12:46.670911 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa1546a4-6408-48b1-99b6-eb424e587575 service cinder] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:46.673316 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-fa1546a4-6408-48b1-99b6-eb424e587575 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:46.710904 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa1546a4-6408-48b1-99b6-eb424e587575 service cinder] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:46.724691 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c096ca3f-514c-4178-a8b9-e9c06c34d1a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:46.725384 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c096ca3f-514c-4178-a8b9-e9c06c34d1a9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:46.725615 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c096ca3f-514c-4178-a8b9-e9c06c34d1a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:46.725912 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c096ca3f-514c-4178-a8b9-e9c06c34d1a9 admin admin] PATH_INFO: `/v3/projects/60b191df1134402f83194ae40d623eb7` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:46.727782 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c096ca3f-514c-4178-a8b9-e9c06c34d1a9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=60b191df1134402f83194ae40d623eb7)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:46.732635 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c096ca3f-514c-4178-a8b9-e9c06c34d1a9 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:46.788955 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:46.793463 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-c096ca3f-514c-4178-a8b9-e9c06c34d1a9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '60b191df1134402f83194ae40d623eb7', 'request_id': 'req-c096ca3f-514c-4178-a8b9-e9c06c34d1a9'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:46.807675 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-c096ca3f-514c-4178-a8b9-e9c06c34d1a9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '60b191df1134402f83194ae40d623eb7', 'request_id': 'req-c096ca3f-514c-4178-a8b9-e9c06c34d1a9'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:46.814204 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-c096ca3f-514c-4178-a8b9-e9c06c34d1a9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '60b191df1134402f83194ae40d623eb7', 'request_id': 'req-c096ca3f-514c-4178-a8b9-e9c06c34d1a9'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:46.825105 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-c096ca3f-514c-4178-a8b9-e9c06c34d1a9 admin admin] The token cache is being invalidate because project 60b191df1134402f83194ae40d623eb7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81624) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:12:46.825510 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-c096ca3f-514c-4178-a8b9-e9c06c34d1a9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:46.826235 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-c096ca3f-514c-4178-a8b9-e9c06c34d1a9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:46.828053 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 949/1921] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:12:46 2022] DELETE /identity/v3/projects/60b191df1134402f83194ae40d623eb7 => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:12:46.832735 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-792dd9a4-9d90-4e3f-a5da-7e035f20c9c7 service neutron] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:46.842508 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-792dd9a4-9d90-4e3f-a5da-7e035f20c9c7 service neutron] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:46.854018 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c096ca3f-514c-4178-a8b9-e9c06c34d1a9 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:46.855752 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-c096ca3f-514c-4178-a8b9-e9c06c34d1a9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:46.857005 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:46.915192 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89b8860d-dae7-40b3-90f9-de550d49772a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:46.915826 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b8860d-dae7-40b3-90f9-de550d49772a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:46.916242 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b8860d-dae7-40b3-90f9-de550d49772a admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:46.916520 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b8860d-dae7-40b3-90f9-de550d49772a admin admin] PATH_INFO: `/v3/users/c37a7697ded84613843c630b4b0ac6a3` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:46.918318 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89b8860d-dae7-40b3-90f9-de550d49772a admin admin] RBAC: Authorizing `identity:delete_user(user_id=c37a7697ded84613843c630b4b0ac6a3)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:46.920287 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89b8860d-dae7-40b3-90f9-de550d49772a admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:46.933962 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c096ca3f-514c-4178-a8b9-e9c06c34d1a9 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:46.940528 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-133b475e-6ff9-432d-8fab-16beaa6ba1a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:46.941166 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-133b475e-6ff9-432d-8fab-16beaa6ba1a9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:46.941442 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-133b475e-6ff9-432d-8fab-16beaa6ba1a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:46.941709 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-133b475e-6ff9-432d-8fab-16beaa6ba1a9 admin admin] PATH_INFO: `/v3/users/f17b1949106d4fd1b0af8bd13b059c6a` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:46.942870 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-133b475e-6ff9-432d-8fab-16beaa6ba1a9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f17b1949106d4fd1b0af8bd13b059c6a)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:46.945034 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-133b475e-6ff9-432d-8fab-16beaa6ba1a9 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:46.978998 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-89b8860d-dae7-40b3-90f9-de550d49772a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c37a7697ded84613843c630b4b0ac6a3'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:46.991861 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-89b8860d-dae7-40b3-90f9-de550d49772a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c37a7697ded84613843c630b4b0ac6a3'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:47.006983 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-133b475e-6ff9-432d-8fab-16beaa6ba1a9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f17b1949106d4fd1b0af8bd13b059c6a'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:47.008015 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-133b475e-6ff9-432d-8fab-16beaa6ba1a9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f17b1949106d4fd1b0af8bd13b059c6a'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:47.008131 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-89b8860d-dae7-40b3-90f9-de550d49772a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c37a7697ded84613843c630b4b0ac6a3'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:47.008808 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-133b475e-6ff9-432d-8fab-16beaa6ba1a9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f17b1949106d4fd1b0af8bd13b059c6a'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:47.016287 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-89b8860d-dae7-40b3-90f9-de550d49772a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c37a7697ded84613843c630b4b0ac6a3'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:47.017034 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-89b8860d-dae7-40b3-90f9-de550d49772a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c37a7697ded84613843c630b4b0ac6a3'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:47.024986 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-133b475e-6ff9-432d-8fab-16beaa6ba1a9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f17b1949106d4fd1b0af8bd13b059c6a'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:47.031130 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 973/1922] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:12:46 2022] DELETE /identity/v3/users/c37a7697ded84613843c630b4b0ac6a3 => generated 0 bytes in 199 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:12:47.036293 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-133b475e-6ff9-432d-8fab-16beaa6ba1a9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f17b1949106d4fd1b0af8bd13b059c6a'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:47.057309 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 950/1923] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:12:46 2022] DELETE /identity/v3/users/f17b1949106d4fd1b0af8bd13b059c6a => generated 0 bytes in 204 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 05:12:47.063085 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c60f979-0dbc-466e-b8f4-3897f22c9cba None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:47.064064 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c60f979-0dbc-466e-b8f4-3897f22c9cba None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:47.064064 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c60f979-0dbc-466e-b8f4-3897f22c9cba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:47.108169 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-6c60f979-0dbc-466e-b8f4-3897f22c9cba None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:12:47.112555 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-6c60f979-0dbc-466e-b8f4-3897f22c9cba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:47.204002 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 974/1924] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:12:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:12:47.214413 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-133b475e-6ff9-432d-8fab-16beaa6ba1a9 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:47.215990 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-133b475e-6ff9-432d-8fab-16beaa6ba1a9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:47.305106 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-133b475e-6ff9-432d-8fab-16beaa6ba1a9 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:47.308690 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8575c926-14fb-4f60-9c33-ba7ba171db71 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:47.309296 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8575c926-14fb-4f60-9c33-ba7ba171db71 service neutron] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:47.309571 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8575c926-14fb-4f60-9c33-ba7ba171db71 service neutron] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:47.309840 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8575c926-14fb-4f60-9c33-ba7ba171db71 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:47.321538 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:47.324347 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8575c926-14fb-4f60-9c33-ba7ba171db71 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:47.326389 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8575c926-14fb-4f60-9c33-ba7ba171db71 service neutron] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:47.338260 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 951/1925] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:12:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 124 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:12:47.436682 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c60f979-0dbc-466e-b8f4-3897f22c9cba None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:47.438256 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-6c60f979-0dbc-466e-b8f4-3897f22c9cba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:47.469780 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c60f979-0dbc-466e-b8f4-3897f22c9cba None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:47.477703 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24b0b17d-7a39-413a-b770-e651b405ff4d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:47.478948 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24b0b17d-7a39-413a-b770-e651b405ff4d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:47.479174 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24b0b17d-7a39-413a-b770-e651b405ff4d admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:47.479596 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24b0b17d-7a39-413a-b770-e651b405ff4d admin admin] PATH_INFO: `/v3/projects/b8d0da5bcfa140c9a9e5e69d1b2ad94c` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:47.481153 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24b0b17d-7a39-413a-b770-e651b405ff4d admin admin] RBAC: Authorizing `identity:delete_project(project_id=b8d0da5bcfa140c9a9e5e69d1b2ad94c)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:47.483968 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24b0b17d-7a39-413a-b770-e651b405ff4d admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:47.545551 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:47.568937 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-24b0b17d-7a39-413a-b770-e651b405ff4d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b8d0da5bcfa140c9a9e5e69d1b2ad94c', 'request_id': 'req-24b0b17d-7a39-413a-b770-e651b405ff4d'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:47.594760 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-24b0b17d-7a39-413a-b770-e651b405ff4d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b8d0da5bcfa140c9a9e5e69d1b2ad94c', 'request_id': 'req-24b0b17d-7a39-413a-b770-e651b405ff4d'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:47.604802 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-24b0b17d-7a39-413a-b770-e651b405ff4d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b8d0da5bcfa140c9a9e5e69d1b2ad94c', 'request_id': 'req-24b0b17d-7a39-413a-b770-e651b405ff4d'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:47.619956 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-24b0b17d-7a39-413a-b770-e651b405ff4d admin admin] The token cache is being invalidate because project b8d0da5bcfa140c9a9e5e69d1b2ad94c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81625) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:12:47.620257 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-24b0b17d-7a39-413a-b770-e651b405ff4d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:47.621089 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-24b0b17d-7a39-413a-b770-e651b405ff4d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:47.626281 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 975/1926] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:12:47 2022] DELETE /identity/v3/projects/b8d0da5bcfa140c9a9e5e69d1b2ad94c => generated 0 bytes in 187 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:12:47.628479 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee093e9-3423-43da-95f7-a662852f216e None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:47.628766 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee093e9-3423-43da-95f7-a662852f216e None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:47.629042 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee093e9-3423-43da-95f7-a662852f216e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:47.670839 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-4ee093e9-3423-43da-95f7-a662852f216e None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:12:47.672085 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-4ee093e9-3423-43da-95f7-a662852f216e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:47.822456 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 952/1927] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:12:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 196 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:12:47.831753 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e27bfbb-a9fc-44b1-bd91-18d0df05edce None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:47.832193 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e27bfbb-a9fc-44b1-bd91-18d0df05edce None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:47.833044 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e27bfbb-a9fc-44b1-bd91-18d0df05edce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:47.887640 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-3e27bfbb-a9fc-44b1-bd91-18d0df05edce None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:12:47.890794 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-3e27bfbb-a9fc-44b1-bd91-18d0df05edce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:47.927302 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 976/1928] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:12:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:12:47.928838 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ee093e9-3423-43da-95f7-a662852f216e None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:47.948236 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ee093e9-3423-43da-95f7-a662852f216e None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:47.962801 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fbc18b1-dc28-4800-9aa4-3cc635eb2f0c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:47.963805 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fbc18b1-dc28-4800-9aa4-3cc635eb2f0c admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:47.964672 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fbc18b1-dc28-4800-9aa4-3cc635eb2f0c admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:47.965485 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fbc18b1-dc28-4800-9aa4-3cc635eb2f0c admin admin] PATH_INFO: `/v3/domains` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:47.967493 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fbc18b1-dc28-4800-9aa4-3cc635eb2f0c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:12:47.967960 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fbc18b1-dc28-4800-9aa4-3cc635eb2f0c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:47.972846 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fbc18b1-dc28-4800-9aa4-3cc635eb2f0c admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:48.046594 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 953/1929] 10.222.0.77 () {58 vars in 1153 bytes} [Tue Aug 30 05:12:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 119 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:12:48.053847 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e27bfbb-a9fc-44b1-bd91-18d0df05edce None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:48.077482 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e27bfbb-a9fc-44b1-bd91-18d0df05edce None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:48.099140 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6fc9191-8926-4d8a-b6f2-a94034db3d69 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:48.101157 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6fc9191-8926-4d8a-b6f2-a94034db3d69 admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:48.102481 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6fc9191-8926-4d8a-b6f2-a94034db3d69 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:48.104003 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6fc9191-8926-4d8a-b6f2-a94034db3d69 admin admin] PATH_INFO: `/v3/projects` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:48.104880 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6fc9191-8926-4d8a-b6f2-a94034db3d69 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:48.113059 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6fc9191-8926-4d8a-b6f2-a94034db3d69 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:48.188235 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fbc18b1-dc28-4800-9aa4-3cc635eb2f0c admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:48.191039 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-2fbc18b1-dc28-4800-9aa4-3cc635eb2f0c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:48.208533 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 977/1930] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:12:48 2022] POST /identity/v3/projects => generated 383 bytes in 157 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:12:48.209624 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6fc9191-8926-4d8a-b6f2-a94034db3d69 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:48.220672 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fbc18b1-dc28-4800-9aa4-3cc635eb2f0c admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:48.317003 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6fc9191-8926-4d8a-b6f2-a94034db3d69 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:48.322572 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2631a497-d83f-4c78-8239-4fcb96295bbf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:48.323245 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2631a497-d83f-4c78-8239-4fcb96295bbf admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:48.323615 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2631a497-d83f-4c78-8239-4fcb96295bbf admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:48.323810 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2631a497-d83f-4c78-8239-4fcb96295bbf admin admin] PATH_INFO: `/v3/users` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:48.324456 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2631a497-d83f-4c78-8239-4fcb96295bbf admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:48.326497 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2631a497-d83f-4c78-8239-4fcb96295bbf admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:48.346319 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87867975-7fbc-4973-ba36-a467220751c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:48.347671 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87867975-7fbc-4973-ba36-a467220751c3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:48.348733 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87867975-7fbc-4973-ba36-a467220751c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:48.349944 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87867975-7fbc-4973-ba36-a467220751c3 admin admin] PATH_INFO: `/v3/projects/b8c7cffa472b424eb9052b47f2bbce8c` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:48.355395 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87867975-7fbc-4973-ba36-a467220751c3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b8c7cffa472b424eb9052b47f2bbce8c)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:48.358852 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87867975-7fbc-4973-ba36-a467220751c3 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:48.375839 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 978/1931] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:12:48 2022] POST /identity/v3/users => generated 292 bytes in 167 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:12:48.378753 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2631a497-d83f-4c78-8239-4fcb96295bbf admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:48.404566 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2631a497-d83f-4c78-8239-4fcb96295bbf admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:48.412949 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e2710fe-30b9-47d2-bc60-d51b9f019998 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:48.414299 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e2710fe-30b9-47d2-bc60-d51b9f019998 admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:48.415000 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e2710fe-30b9-47d2-bc60-d51b9f019998 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:48.416404 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e2710fe-30b9-47d2-bc60-d51b9f019998 admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:48.418954 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e2710fe-30b9-47d2-bc60-d51b9f019998 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:48.424059 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e2710fe-30b9-47d2-bc60-d51b9f019998 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:48.458487 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 979/1932] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:12:48 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:12:48.460056 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e2710fe-30b9-47d2-bc60-d51b9f019998 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:48.480439 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e2710fe-30b9-47d2-bc60-d51b9f019998 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:48.488847 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-87867975-7fbc-4973-ba36-a467220751c3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b8c7cffa472b424eb9052b47f2bbce8c', 'request_id': 'req-87867975-7fbc-4973-ba36-a467220751c3'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:48.510509 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-87867975-7fbc-4973-ba36-a467220751c3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b8c7cffa472b424eb9052b47f2bbce8c', 'request_id': 'req-87867975-7fbc-4973-ba36-a467220751c3'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:48.521576 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-87867975-7fbc-4973-ba36-a467220751c3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b8c7cffa472b424eb9052b47f2bbce8c', 'request_id': 'req-87867975-7fbc-4973-ba36-a467220751c3'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:48.545738 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-87867975-7fbc-4973-ba36-a467220751c3 admin admin] The token cache is being invalidate because project b8c7cffa472b424eb9052b47f2bbce8c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81624) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:12:48.546291 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-87867975-7fbc-4973-ba36-a467220751c3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:48.547672 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-87867975-7fbc-4973-ba36-a467220751c3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:48.550597 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 954/1933] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:12:48 2022] DELETE /identity/v3/projects/b8c7cffa472b424eb9052b47f2bbce8c => generated 0 bytes in 363 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:12:48.554834 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87867975-7fbc-4973-ba36-a467220751c3 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:48.556653 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-87867975-7fbc-4973-ba36-a467220751c3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:48.567872 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d66616d3-4c05-44a8-bd72-c96a79c063ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:48.569453 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d66616d3-4c05-44a8-bd72-c96a79c063ea admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:48.569866 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d66616d3-4c05-44a8-bd72-c96a79c063ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:48.570211 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d66616d3-4c05-44a8-bd72-c96a79c063ea admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:48.570995 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d66616d3-4c05-44a8-bd72-c96a79c063ea admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:48.573575 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d66616d3-4c05-44a8-bd72-c96a79c063ea admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:48.573749 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:48.585819 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 980/1934] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:12:48 2022] GET /identity/v3/roles => generated 1420 bytes in 127 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:12:48.594902 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d66616d3-4c05-44a8-bd72-c96a79c063ea admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:48.594902 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-d66616d3-4c05-44a8-bd72-c96a79c063ea admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:48.608786 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=90,keystone:DELETE=5,keystone:INSERT=5 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:48.613374 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d66616d3-4c05-44a8-bd72-c96a79c063ea admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:48.619437 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa08c150-6f07-4d21-a371-188fdc946ead admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:48.620097 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa08c150-6f07-4d21-a371-188fdc946ead admin admin] REQUEST_METHOD: `PUT` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:48.620709 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa08c150-6f07-4d21-a371-188fdc946ead admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:48.621301 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa08c150-6f07-4d21-a371-188fdc946ead admin admin] PATH_INFO: `/v3/projects/7884824e60d248b683b215d10aa3fadf/users/f3358a3a394f48838bd35aeed77b5d88/roles/6f03d86980c34c36b44a8d6764ff4fda` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:48.636140 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa08c150-6f07-4d21-a371-188fdc946ead admin admin] RBAC: Authorizing `identity:create_grant(project_id=7884824e60d248b683b215d10aa3fadf, user_id=f3358a3a394f48838bd35aeed77b5d88, role_id=6f03d86980c34c36b44a8d6764ff4fda)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:48.638781 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa08c150-6f07-4d21-a371-188fdc946ead admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:48.641144 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87867975-7fbc-4973-ba36-a467220751c3 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:48.647035 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6fe58ac-5b04-4718-ab6c-c5aa528121aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:48.647662 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6fe58ac-5b04-4718-ab6c-c5aa528121aa admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:48.647954 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6fe58ac-5b04-4718-ab6c-c5aa528121aa admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:48.648257 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6fe58ac-5b04-4718-ab6c-c5aa528121aa admin admin] PATH_INFO: `/v3/users/863102c4f8c04ffa825ce2e0cae463d5` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:48.649649 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6fe58ac-5b04-4718-ab6c-c5aa528121aa admin admin] RBAC: Authorizing `identity:delete_user(user_id=863102c4f8c04ffa825ce2e0cae463d5)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:48.651604 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6fe58ac-5b04-4718-ab6c-c5aa528121aa admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:48.654907 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 981/1935] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:12:48 2022] PUT /identity/v3/projects/7884824e60d248b683b215d10aa3fadf/users/f3358a3a394f48838bd35aeed77b5d88/roles/6f03d86980c34c36b44a8d6764ff4fda => generated 0 bytes in 65 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:12:48.665811 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b5afe4-b4af-413d-b431-6e8f73cbff6a None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:48.666131 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b5afe4-b4af-413d-b431-6e8f73cbff6a None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:48.666410 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b5afe4-b4af-413d-b431-6e8f73cbff6a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:48.695475 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-99b5afe4-b4af-413d-b431-6e8f73cbff6a None None] MFA Rules not processed for user `f3358a3a394f48838bd35aeed77b5d88`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:12:48.696760 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-99b5afe4-b4af-413d-b431-6e8f73cbff6a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:48.698697 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-d6fe58ac-5b04-4718-ab6c-c5aa528121aa admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '863102c4f8c04ffa825ce2e0cae463d5'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:48.699585 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-d6fe58ac-5b04-4718-ab6c-c5aa528121aa admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '863102c4f8c04ffa825ce2e0cae463d5'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:48.700205 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-d6fe58ac-5b04-4718-ab6c-c5aa528121aa admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '863102c4f8c04ffa825ce2e0cae463d5'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:48.712388 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-d6fe58ac-5b04-4718-ab6c-c5aa528121aa admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '863102c4f8c04ffa825ce2e0cae463d5'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:48.719651 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-d6fe58ac-5b04-4718-ab6c-c5aa528121aa admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '863102c4f8c04ffa825ce2e0cae463d5'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:48.733703 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 955/1936] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:12:48 2022] DELETE /identity/v3/users/863102c4f8c04ffa825ce2e0cae463d5 => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:12:48.775603 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 982/1937] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:12:48 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 05:12:48.776703 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6fe58ac-5b04-4718-ab6c-c5aa528121aa admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:48.778842 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-d6fe58ac-5b04-4718-ab6c-c5aa528121aa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:48.854955 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6fe58ac-5b04-4718-ab6c-c5aa528121aa admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:48.859581 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71800297-c871-496f-8de2-f58f131a76d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:48.860189 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71800297-c871-496f-8de2-f58f131a76d7 admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:48.860460 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71800297-c871-496f-8de2-f58f131a76d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:48.860731 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71800297-c871-496f-8de2-f58f131a76d7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:48.861357 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71800297-c871-496f-8de2-f58f131a76d7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:48.863821 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71800297-c871-496f-8de2-f58f131a76d7 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:48.902178 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 956/1938] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:12:48 2022] POST /identity/v3/projects => generated 383 bytes in 123 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 05:12:48.903440 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99b5afe4-b4af-413d-b431-6e8f73cbff6a None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:48.981215 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99b5afe4-b4af-413d-b431-6e8f73cbff6a None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:49.035217 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71800297-c871-496f-8de2-f58f131a76d7 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:49.037250 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-71800297-c871-496f-8de2-f58f131a76d7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:49.063908 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6566248f-0efc-4adf-8774-11a860c9c5e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:49.066528 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6566248f-0efc-4adf-8774-11a860c9c5e7 admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:49.066903 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6566248f-0efc-4adf-8774-11a860c9c5e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:49.067386 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6566248f-0efc-4adf-8774-11a860c9c5e7 admin admin] PATH_INFO: `/v3/users` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:49.068893 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6566248f-0efc-4adf-8774-11a860c9c5e7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:49.071122 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6566248f-0efc-4adf-8774-11a860c9c5e7 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:49.103390 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:49.123065 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 983/1939] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:12:48 2022] POST /identity/v3/users => generated 298 bytes in 217 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:12:49.125053 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6566248f-0efc-4adf-8774-11a860c9c5e7 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:49.155231 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6566248f-0efc-4adf-8774-11a860c9c5e7 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:49.166688 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbeeeb72-9135-4603-8f73-aa219f86e9c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:49.167429 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbeeeb72-9135-4603-8f73-aa219f86e9c2 admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:49.168153 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbeeeb72-9135-4603-8f73-aa219f86e9c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:49.169238 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbeeeb72-9135-4603-8f73-aa219f86e9c2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:49.171709 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbeeeb72-9135-4603-8f73-aa219f86e9c2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:49.173497 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbeeeb72-9135-4603-8f73-aa219f86e9c2 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:49.177302 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:49.200803 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 984/1940] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:12:49 2022] GET /identity/v3/roles => generated 1420 bytes in 76 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:12:49.201900 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71800297-c871-496f-8de2-f58f131a76d7 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:49.204758 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbeeeb72-9135-4603-8f73-aa219f86e9c2 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:49.221158 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df0f1ddf-29a0-49a8-8634-51eceff78199 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:49.222453 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df0f1ddf-29a0-49a8-8634-51eceff78199 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:49.222798 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df0f1ddf-29a0-49a8-8634-51eceff78199 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:49.223330 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df0f1ddf-29a0-49a8-8634-51eceff78199 admin admin] PATH_INFO: `/v3/projects/2c5193813eb045d5910534d3c6e6234b` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:49.225602 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbeeeb72-9135-4603-8f73-aa219f86e9c2 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:49.227412 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df0f1ddf-29a0-49a8-8634-51eceff78199 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2c5193813eb045d5910534d3c6e6234b)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:49.229410 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df0f1ddf-29a0-49a8-8634-51eceff78199 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:49.240779 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfbdb968-d270-4be7-ba02-add7bc9c844c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:49.241553 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfbdb968-d270-4be7-ba02-add7bc9c844c admin admin] REQUEST_METHOD: `PUT` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:49.241898 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfbdb968-d270-4be7-ba02-add7bc9c844c admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:49.242261 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfbdb968-d270-4be7-ba02-add7bc9c844c admin admin] PATH_INFO: `/v3/projects/5d156e26b9ac4afa88354d61756f2068/users/7f94c03417f14d2dbdd70bf158b6c4d3/roles/9d9d5d2c270146488249b7ed943280db` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:49.278014 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfbdb968-d270-4be7-ba02-add7bc9c844c admin admin] RBAC: Authorizing `identity:create_grant(project_id=5d156e26b9ac4afa88354d61756f2068, user_id=7f94c03417f14d2dbdd70bf158b6c4d3, role_id=9d9d5d2c270146488249b7ed943280db)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:49.280914 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfbdb968-d270-4be7-ba02-add7bc9c844c admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:49.315351 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 985/1941] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:12:49 2022] PUT /identity/v3/projects/5d156e26b9ac4afa88354d61756f2068/users/7f94c03417f14d2dbdd70bf158b6c4d3/roles/9d9d5d2c270146488249b7ed943280db => generated 0 bytes in 108 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 05:12:49.316999 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfbdb968-d270-4be7-ba02-add7bc9c844c admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:49.349466 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-df0f1ddf-29a0-49a8-8634-51eceff78199 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2c5193813eb045d5910534d3c6e6234b', 'request_id': 'req-df0f1ddf-29a0-49a8-8634-51eceff78199'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:49.359584 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-df0f1ddf-29a0-49a8-8634-51eceff78199 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2c5193813eb045d5910534d3c6e6234b', 'request_id': 'req-df0f1ddf-29a0-49a8-8634-51eceff78199'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:49.370254 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-df0f1ddf-29a0-49a8-8634-51eceff78199 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2c5193813eb045d5910534d3c6e6234b', 'request_id': 'req-df0f1ddf-29a0-49a8-8634-51eceff78199'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:49.392572 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-df0f1ddf-29a0-49a8-8634-51eceff78199 admin admin] The token cache is being invalidate because project 2c5193813eb045d5910534d3c6e6234b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81624) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:12:49.392872 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-df0f1ddf-29a0-49a8-8634-51eceff78199 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:49.393633 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-df0f1ddf-29a0-49a8-8634-51eceff78199 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:49.395953 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 957/1942] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:12:49 2022] DELETE /identity/v3/projects/2c5193813eb045d5910534d3c6e6234b => generated 0 bytes in 362 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 05:12:49.403917 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-135f909c-b9f0-479f-a417-fcf5787c7765 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:49.404488 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-135f909c-b9f0-479f-a417-fcf5787c7765 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:49.405090 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-135f909c-b9f0-479f-a417-fcf5787c7765 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:49.434947 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-135f909c-b9f0-479f-a417-fcf5787c7765 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:12:49.437247 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-135f909c-b9f0-479f-a417-fcf5787c7765 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:49.441862 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfbdb968-d270-4be7-ba02-add7bc9c844c admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:49.537629 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3afee512-b5fc-4c5e-a1a2-3e7bc1047ab3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:49.538480 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3afee512-b5fc-4c5e-a1a2-3e7bc1047ab3 admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:49.538992 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3afee512-b5fc-4c5e-a1a2-3e7bc1047ab3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:49.539485 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3afee512-b5fc-4c5e-a1a2-3e7bc1047ab3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:49.540631 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3afee512-b5fc-4c5e-a1a2-3e7bc1047ab3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:49.543147 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3afee512-b5fc-4c5e-a1a2-3e7bc1047ab3 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:49.552389 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 958/1943] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:12:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:12:49.553966 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20bddf07-2ef5-4f94-be5d-bb4d7599077f None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:49.554315 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20bddf07-2ef5-4f94-be5d-bb4d7599077f None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:49.554726 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20bddf07-2ef5-4f94-be5d-bb4d7599077f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:49.571254 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 986/1944] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:12:49 2022] GET /identity/v3/roles => generated 1420 bytes in 255 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:12:49.572175 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3afee512-b5fc-4c5e-a1a2-3e7bc1047ab3 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:49.579342 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-3afee512-b5fc-4c5e-a1a2-3e7bc1047ab3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:49.593299 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-20bddf07-2ef5-4f94-be5d-bb4d7599077f None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:12:49.595838 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-20bddf07-2ef5-4f94-be5d-bb4d7599077f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:49.595985 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3afee512-b5fc-4c5e-a1a2-3e7bc1047ab3 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:49.602596 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37d583a4-a8c1-48fe-ac16-fb4ec82309d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:49.603314 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37d583a4-a8c1-48fe-ac16-fb4ec82309d9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:49.603596 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37d583a4-a8c1-48fe-ac16-fb4ec82309d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:49.604035 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37d583a4-a8c1-48fe-ac16-fb4ec82309d9 admin admin] PATH_INFO: `/v3/projects/5d156e26b9ac4afa88354d61756f2068/users/7f94c03417f14d2dbdd70bf158b6c4d3/roles/9d9d5d2c270146488249b7ed943280db` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:49.606429 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37d583a4-a8c1-48fe-ac16-fb4ec82309d9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5d156e26b9ac4afa88354d61756f2068, user_id=7f94c03417f14d2dbdd70bf158b6c4d3, role_id=9d9d5d2c270146488249b7ed943280db)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:49.609606 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37d583a4-a8c1-48fe-ac16-fb4ec82309d9 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:49.623399 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 987/1945] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:12:49 2022] PUT /identity/v3/projects/5d156e26b9ac4afa88354d61756f2068/users/7f94c03417f14d2dbdd70bf158b6c4d3/roles/9d9d5d2c270146488249b7ed943280db => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:12:49.629631 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 959/1946] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:12:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 77 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:12:49.633823 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37d583a4-a8c1-48fe-ac16-fb4ec82309d9 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:49.639990 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c745ab-165e-4824-b417-18a8cd5e257d None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:49.640358 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c745ab-165e-4824-b417-18a8cd5e257d None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:49.640773 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c745ab-165e-4824-b417-18a8cd5e257d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:49.688815 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-91c745ab-165e-4824-b417-18a8cd5e257d None None] MFA Rules not processed for user `7f94c03417f14d2dbdd70bf158b6c4d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:12:49.690253 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-91c745ab-165e-4824-b417-18a8cd5e257d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:49.742448 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37d583a4-a8c1-48fe-ac16-fb4ec82309d9 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:49.771135 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97da7665-d148-45c2-bed1-d8c3834d5e5c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:49.771776 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97da7665-d148-45c2-bed1-d8c3834d5e5c admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:49.772028 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97da7665-d148-45c2-bed1-d8c3834d5e5c admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:49.772364 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97da7665-d148-45c2-bed1-d8c3834d5e5c admin admin] PATH_INFO: `/v3/domains` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:49.772928 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97da7665-d148-45c2-bed1-d8c3834d5e5c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:12:49.773248 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97da7665-d148-45c2-bed1-d8c3834d5e5c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:49.774853 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97da7665-d148-45c2-bed1-d8c3834d5e5c admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:49.783294 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:49.788495 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:49.809400 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 988/1947] 10.222.0.77 () {58 vars in 1153 bytes} [Tue Aug 30 05:12:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 05:12:49.815408 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97da7665-d148-45c2-bed1-d8c3834d5e5c admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:49.838106 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97da7665-d148-45c2-bed1-d8c3834d5e5c admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:49.853486 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0bab208-5d78-4138-b882-bb7d94e8fa70 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:49.854205 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0bab208-5d78-4138-b882-bb7d94e8fa70 admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:49.854560 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0bab208-5d78-4138-b882-bb7d94e8fa70 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:49.854930 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0bab208-5d78-4138-b882-bb7d94e8fa70 admin admin] PATH_INFO: `/v3/projects` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:49.856648 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0bab208-5d78-4138-b882-bb7d94e8fa70 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:49.859812 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0bab208-5d78-4138-b882-bb7d94e8fa70 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:49.896047 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 960/1948] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:12:49 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 259 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:12:49.948484 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 989/1949] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:12:49 2022] POST /identity/v3/projects => generated 367 bytes in 133 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:12:49.952859 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91c745ab-165e-4824-b417-18a8cd5e257d None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:49.970353 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91c745ab-165e-4824-b417-18a8cd5e257d None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:49.993661 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0bab208-5d78-4138-b882-bb7d94e8fa70 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:49.995721 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-a0bab208-5d78-4138-b882-bb7d94e8fa70 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:50.108971 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0bab208-5d78-4138-b882-bb7d94e8fa70 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:50.120943 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25be8538-8904-4036-aac0-0a328ce8e5bf service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '019a79b2affe478b8a30d1c538d0415c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:50.121843 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25be8538-8904-4036-aac0-0a328ce8e5bf service cinder] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:50.122180 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25be8538-8904-4036-aac0-0a328ce8e5bf service cinder] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:50.122678 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25be8538-8904-4036-aac0-0a328ce8e5bf service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:50.171099 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25dd8532-7441-4395-81fc-e9ff92ef7802 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:50.173166 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25dd8532-7441-4395-81fc-e9ff92ef7802 admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:50.175570 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25dd8532-7441-4395-81fc-e9ff92ef7802 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:50.176030 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25dd8532-7441-4395-81fc-e9ff92ef7802 admin admin] PATH_INFO: `/v3/users` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:50.177348 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25dd8532-7441-4395-81fc-e9ff92ef7802 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:50.185452 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25dd8532-7441-4395-81fc-e9ff92ef7802 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:50.209950 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25be8538-8904-4036-aac0-0a328ce8e5bf service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:50.212734 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25be8538-8904-4036-aac0-0a328ce8e5bf service cinder] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:50.231796 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 961/1950] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:12:49 2022] POST /identity/v3/users => generated 284 bytes in 279 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:12:50.237687 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25dd8532-7441-4395-81fc-e9ff92ef7802 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:50.243873 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 990/1951] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:12:49 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 251 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 05:12:50.255256 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25dd8532-7441-4395-81fc-e9ff92ef7802 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:50.262138 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec32d309-3d0e-4bfc-9db4-be12f81182c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:50.263413 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec32d309-3d0e-4bfc-9db4-be12f81182c6 admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:50.263996 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec32d309-3d0e-4bfc-9db4-be12f81182c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:50.264272 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec32d309-3d0e-4bfc-9db4-be12f81182c6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:50.264933 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec32d309-3d0e-4bfc-9db4-be12f81182c6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:50.266553 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec32d309-3d0e-4bfc-9db4-be12f81182c6 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:50.279486 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 962/1952] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:12:50 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 05:12:50.289825 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25be8538-8904-4036-aac0-0a328ce8e5bf service cinder] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:50.310560 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25be8538-8904-4036-aac0-0a328ce8e5bf service cinder] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:50.311146 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02413120-8cfe-4948-b065-249536027806 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:50.312018 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02413120-8cfe-4948-b065-249536027806 admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:50.312308 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02413120-8cfe-4948-b065-249536027806 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:50.312624 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02413120-8cfe-4948-b065-249536027806 admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:50.313544 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02413120-8cfe-4948-b065-249536027806 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:50.315427 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02413120-8cfe-4948-b065-249536027806 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:50.330515 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 991/1953] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:12:50 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:12:50.333380 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec32d309-3d0e-4bfc-9db4-be12f81182c6 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:50.344239 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec32d309-3d0e-4bfc-9db4-be12f81182c6 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:50.350602 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88c015c5-24b6-4ac5-a229-8027ebccf141 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:50.351295 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88c015c5-24b6-4ac5-a229-8027ebccf141 admin admin] REQUEST_METHOD: `PUT` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:50.351578 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88c015c5-24b6-4ac5-a229-8027ebccf141 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:50.351866 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88c015c5-24b6-4ac5-a229-8027ebccf141 admin admin] PATH_INFO: `/v3/projects/d6884877d69d43b4a31ff6a2bee7f747/users/8795355dcaa14d30954ef3739d32e1dc/roles/6f03d86980c34c36b44a8d6764ff4fda` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:50.371125 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88c015c5-24b6-4ac5-a229-8027ebccf141 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d6884877d69d43b4a31ff6a2bee7f747, user_id=8795355dcaa14d30954ef3739d32e1dc, role_id=6f03d86980c34c36b44a8d6764ff4fda)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:50.374585 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88c015c5-24b6-4ac5-a229-8027ebccf141 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:50.375333 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02413120-8cfe-4948-b065-249536027806 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:50.389513 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02413120-8cfe-4948-b065-249536027806 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:50.403695 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 963/1954] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:12:50 2022] PUT /identity/v3/projects/d6884877d69d43b4a31ff6a2bee7f747/users/8795355dcaa14d30954ef3739d32e1dc/roles/6f03d86980c34c36b44a8d6764ff4fda => generated 0 bytes in 63 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:12:50.406808 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb1d1ece-e761-4fcd-b720-2b6d85cc8b09 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:50.407140 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb1d1ece-e761-4fcd-b720-2b6d85cc8b09 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:50.407417 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb1d1ece-e761-4fcd-b720-2b6d85cc8b09 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:50.442855 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-fb1d1ece-e761-4fcd-b720-2b6d85cc8b09 None None] MFA Rules not processed for user `8795355dcaa14d30954ef3739d32e1dc`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:12:50.444390 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-fb1d1ece-e761-4fcd-b720-2b6d85cc8b09 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:50.476651 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b887ef96-1e26-41ce-b4c8-727de607a972 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '019a79b2affe478b8a30d1c538d0415c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:50.477265 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b887ef96-1e26-41ce-b4c8-727de607a972 service cinder] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:50.477766 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b887ef96-1e26-41ce-b4c8-727de607a972 service cinder] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:50.477904 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b887ef96-1e26-41ce-b4c8-727de607a972 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:50.479570 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-b887ef96-1e26-41ce-b4c8-727de607a972 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:50.508621 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:50.529850 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 964/1955] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:12:50 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:12:50.535231 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb1d1ece-e761-4fcd-b720-2b6d85cc8b09 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:50.547777 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:50.551334 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b887ef96-1e26-41ce-b4c8-727de607a972 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:50.553057 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b887ef96-1e26-41ce-b4c8-727de607a972 service cinder] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:50.564516 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 992/1956] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:12:50 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 190 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 05:12:50.620679 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb1d1ece-e761-4fcd-b720-2b6d85cc8b09 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:50.625726 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5cb9f85-80c6-42e1-83fe-e3ba40fbbaae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:50.626347 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5cb9f85-80c6-42e1-83fe-e3ba40fbbaae admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:50.626631 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5cb9f85-80c6-42e1-83fe-e3ba40fbbaae admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:50.626937 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5cb9f85-80c6-42e1-83fe-e3ba40fbbaae admin admin] PATH_INFO: `/v3/projects` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:50.627607 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5cb9f85-80c6-42e1-83fe-e3ba40fbbaae admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:50.629550 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5cb9f85-80c6-42e1-83fe-e3ba40fbbaae admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:50.662873 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 965/1957] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:12:50 2022] POST /identity/v3/projects => generated 369 bytes in 129 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 05:12:50.667063 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b887ef96-1e26-41ce-b4c8-727de607a972 service cinder] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:50.677775 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b887ef96-1e26-41ce-b4c8-727de607a972 service cinder] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:50.738423 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51ccbcd9-ade9-410c-a20e-26d86e432dae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:50.739072 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51ccbcd9-ade9-410c-a20e-26d86e432dae admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:50.739348 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51ccbcd9-ade9-410c-a20e-26d86e432dae admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:50.739612 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51ccbcd9-ade9-410c-a20e-26d86e432dae admin admin] PATH_INFO: `/v3/users` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:50.740228 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51ccbcd9-ade9-410c-a20e-26d86e432dae admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:50.742123 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51ccbcd9-ade9-410c-a20e-26d86e432dae admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:50.764574 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 993/1958] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:12:50 2022] POST /identity/v3/users => generated 291 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:12:50.769648 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5cb9f85-80c6-42e1-83fe-e3ba40fbbaae admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:50.779915 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5cb9f85-80c6-42e1-83fe-e3ba40fbbaae admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:50.786692 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a5d73f3-dcc2-4c09-9563-9087e1e9c151 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:50.787416 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a5d73f3-dcc2-4c09-9563-9087e1e9c151 admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:50.787833 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a5d73f3-dcc2-4c09-9563-9087e1e9c151 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:50.788113 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a5d73f3-dcc2-4c09-9563-9087e1e9c151 admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:50.788871 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a5d73f3-dcc2-4c09-9563-9087e1e9c151 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:50.790498 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a5d73f3-dcc2-4c09-9563-9087e1e9c151 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:50.801575 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 966/1959] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:12:50 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:12:50.805667 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51ccbcd9-ade9-410c-a20e-26d86e432dae admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:50.826089 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51ccbcd9-ade9-410c-a20e-26d86e432dae admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:50.826089 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6217cfb3-eb5c-4cdf-ba61-0e361416ec1a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:50.826089 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6217cfb3-eb5c-4cdf-ba61-0e361416ec1a admin admin] REQUEST_METHOD: `PUT` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:50.826089 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6217cfb3-eb5c-4cdf-ba61-0e361416ec1a admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:50.826446 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6217cfb3-eb5c-4cdf-ba61-0e361416ec1a admin admin] PATH_INFO: `/v3/projects/3079e3ce4f124d4591f17f78e271d29c/users/425ed8189fc247b4a72dd6b1ef8bf85c/roles/9d9d5d2c270146488249b7ed943280db` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:50.843820 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6217cfb3-eb5c-4cdf-ba61-0e361416ec1a admin admin] RBAC: Authorizing `identity:create_grant(project_id=3079e3ce4f124d4591f17f78e271d29c, user_id=425ed8189fc247b4a72dd6b1ef8bf85c, role_id=9d9d5d2c270146488249b7ed943280db)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:50.847148 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6217cfb3-eb5c-4cdf-ba61-0e361416ec1a admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:50.861835 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 994/1960] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:12:50 2022] PUT /identity/v3/projects/3079e3ce4f124d4591f17f78e271d29c/users/425ed8189fc247b4a72dd6b1ef8bf85c/roles/9d9d5d2c270146488249b7ed943280db => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:12:50.865553 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a5d73f3-dcc2-4c09-9563-9087e1e9c151 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:50.876604 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a5d73f3-dcc2-4c09-9563-9087e1e9c151 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:50.945757 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc1ca015-6df7-4c1a-ad2f-63b24475e767 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:50.946362 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc1ca015-6df7-4c1a-ad2f-63b24475e767 admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:50.946640 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc1ca015-6df7-4c1a-ad2f-63b24475e767 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:50.946940 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc1ca015-6df7-4c1a-ad2f-63b24475e767 admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:50.947575 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc1ca015-6df7-4c1a-ad2f-63b24475e767 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:50.949148 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc1ca015-6df7-4c1a-ad2f-63b24475e767 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:50.959755 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 967/1961] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:12:50 2022] GET /identity/v3/roles => generated 1420 bytes in 95 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:12:50.963919 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6217cfb3-eb5c-4cdf-ba61-0e361416ec1a admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:50.975575 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6217cfb3-eb5c-4cdf-ba61-0e361416ec1a admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:50.981585 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52af333d-16b8-4c38-a289-2d3f52626352 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:50.982263 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52af333d-16b8-4c38-a289-2d3f52626352 admin admin] REQUEST_METHOD: `PUT` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:50.982539 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52af333d-16b8-4c38-a289-2d3f52626352 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:50.982807 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52af333d-16b8-4c38-a289-2d3f52626352 admin admin] PATH_INFO: `/v3/projects/3079e3ce4f124d4591f17f78e271d29c/users/425ed8189fc247b4a72dd6b1ef8bf85c/roles/9d9d5d2c270146488249b7ed943280db` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:50.984731 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52af333d-16b8-4c38-a289-2d3f52626352 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3079e3ce4f124d4591f17f78e271d29c, user_id=425ed8189fc247b4a72dd6b1ef8bf85c, role_id=9d9d5d2c270146488249b7ed943280db)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:50.987450 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52af333d-16b8-4c38-a289-2d3f52626352 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:50.998137 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 995/1962] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:12:50 2022] PUT /identity/v3/projects/3079e3ce4f124d4591f17f78e271d29c/users/425ed8189fc247b4a72dd6b1ef8bf85c/roles/9d9d5d2c270146488249b7ed943280db => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:12:51.010579 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7e94d1d-93be-4b82-84c1-161e3421fe38 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:51.010955 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7e94d1d-93be-4b82-84c1-161e3421fe38 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:51.011149 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7e94d1d-93be-4b82-84c1-161e3421fe38 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:51.040686 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-b7e94d1d-93be-4b82-84c1-161e3421fe38 None None] MFA Rules not processed for user `425ed8189fc247b4a72dd6b1ef8bf85c`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:12:51.041900 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-b7e94d1d-93be-4b82-84c1-161e3421fe38 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:51.139107 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 968/1963] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:12:51 2022] POST /identity/v3/auth/tokens => generated 3313 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:12:51.142799 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52af333d-16b8-4c38-a289-2d3f52626352 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:51.146417 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-52af333d-16b8-4c38-a289-2d3f52626352 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:51.155782 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7e94d1d-93be-4b82-84c1-161e3421fe38 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:51.161282 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:51.231100 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7e94d1d-93be-4b82-84c1-161e3421fe38 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:51.237997 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52af333d-16b8-4c38-a289-2d3f52626352 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:51.240421 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e02d2bed-cb44-4ecc-9007-efe57e5e90ae service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '019a79b2affe478b8a30d1c538d0415c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:51.241033 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e02d2bed-cb44-4ecc-9007-efe57e5e90ae service cinder] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:51.241315 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e02d2bed-cb44-4ecc-9007-efe57e5e90ae service cinder] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:51.241588 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e02d2bed-cb44-4ecc-9007-efe57e5e90ae service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:51.242674 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12d32f13-8b29-42db-bce3-c33bba3d68ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:51.243323 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12d32f13-8b29-42db-bce3-c33bba3d68ab admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:51.243605 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12d32f13-8b29-42db-bce3-c33bba3d68ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:51.243877 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12d32f13-8b29-42db-bce3-c33bba3d68ab admin admin] PATH_INFO: `/v3/users/9372a0808794476885c8e73c97387970` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:51.244983 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12d32f13-8b29-42db-bce3-c33bba3d68ab admin admin] RBAC: Authorizing `identity:delete_user(user_id=9372a0808794476885c8e73c97387970)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:51.246928 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12d32f13-8b29-42db-bce3-c33bba3d68ab admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:51.251730 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e02d2bed-cb44-4ecc-9007-efe57e5e90ae service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:51.253732 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e02d2bed-cb44-4ecc-9007-efe57e5e90ae service cinder] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:51.263480 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 969/1964] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:12:51 2022] GET /identity/v3/auth/tokens => generated 3313 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 05:12:51.270096 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=92,keystone:INSERT=5,keystone:DELETE=3 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:51.306922 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-12d32f13-8b29-42db-bce3-c33bba3d68ab admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9372a0808794476885c8e73c97387970'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:51.310885 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-12d32f13-8b29-42db-bce3-c33bba3d68ab admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9372a0808794476885c8e73c97387970'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:51.314344 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e02d2bed-cb44-4ecc-9007-efe57e5e90ae service cinder] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:51.316607 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-e02d2bed-cb44-4ecc-9007-efe57e5e90ae service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:51.324318 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-12d32f13-8b29-42db-bce3-c33bba3d68ab admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9372a0808794476885c8e73c97387970'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:51.327605 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e02d2bed-cb44-4ecc-9007-efe57e5e90ae service cinder] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:51.331322 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2946089-f559-467a-b423-f5110426551c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '019a79b2affe478b8a30d1c538d0415c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:51.331486 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-12d32f13-8b29-42db-bce3-c33bba3d68ab admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9372a0808794476885c8e73c97387970'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:51.331911 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2946089-f559-467a-b423-f5110426551c service cinder] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:51.332147 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-12d32f13-8b29-42db-bce3-c33bba3d68ab admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9372a0808794476885c8e73c97387970'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:51.332245 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2946089-f559-467a-b423-f5110426551c service cinder] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:51.332468 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2946089-f559-467a-b423-f5110426551c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:51.334584 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-c2946089-f559-467a-b423-f5110426551c service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:51.338333 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 996/1965] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:12:51 2022] DELETE /identity/v3/users/9372a0808794476885c8e73c97387970 => generated 0 bytes in 196 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 05:12:51.342540 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2316401d-059d-486e-b0d5-c30d4fa675dd None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:51.342847 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2316401d-059d-486e-b0d5-c30d4fa675dd None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:51.343128 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2316401d-059d-486e-b0d5-c30d4fa675dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:51.361548 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-2316401d-059d-486e-b0d5-c30d4fa675dd None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:12:51.363298 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-2316401d-059d-486e-b0d5-c30d4fa675dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:51.396558 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2946089-f559-467a-b423-f5110426551c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:51.398014 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2946089-f559-467a-b423-f5110426551c service cinder] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:51.407714 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 970/1966] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:12:51 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 05:12:51.421532 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 997/1967] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:12:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 05:12:51.431359 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2946089-f559-467a-b423-f5110426551c service cinder] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:51.432744 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-c2946089-f559-467a-b423-f5110426551c service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:51.499506 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2946089-f559-467a-b423-f5110426551c service cinder] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:51.550044 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a78f171a-b14d-4785-b56c-4b0cb920d0f1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:51.550647 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a78f171a-b14d-4785-b56c-4b0cb920d0f1 service neutron] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:51.550964 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a78f171a-b14d-4785-b56c-4b0cb920d0f1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:51.551224 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a78f171a-b14d-4785-b56c-4b0cb920d0f1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:51.561395 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a78f171a-b14d-4785-b56c-4b0cb920d0f1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:51.562844 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a78f171a-b14d-4785-b56c-4b0cb920d0f1 service neutron] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:51.572348 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 971/1968] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:12:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 141 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:12:51.870377 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2316401d-059d-486e-b0d5-c30d4fa675dd None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:51.871877 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-2316401d-059d-486e-b0d5-c30d4fa675dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:51.886228 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2316401d-059d-486e-b0d5-c30d4fa675dd None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:51.891488 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5c911b3-a370-4d88-862b-130a3e6067e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:51.892007 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5c911b3-a370-4d88-862b-130a3e6067e6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:51.892290 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5c911b3-a370-4d88-862b-130a3e6067e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:51.892563 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5c911b3-a370-4d88-862b-130a3e6067e6 admin admin] PATH_INFO: `/v3/projects/f84f3c42dcc54722b46789217a4efd66` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:51.893772 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5c911b3-a370-4d88-862b-130a3e6067e6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f84f3c42dcc54722b46789217a4efd66)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:51.895740 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5c911b3-a370-4d88-862b-130a3e6067e6 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:51.975134 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-d5c911b3-a370-4d88-862b-130a3e6067e6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f84f3c42dcc54722b46789217a4efd66', 'request_id': 'req-d5c911b3-a370-4d88-862b-130a3e6067e6'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:51.986376 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-d5c911b3-a370-4d88-862b-130a3e6067e6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f84f3c42dcc54722b46789217a4efd66', 'request_id': 'req-d5c911b3-a370-4d88-862b-130a3e6067e6'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:52.004892 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-d5c911b3-a370-4d88-862b-130a3e6067e6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f84f3c42dcc54722b46789217a4efd66', 'request_id': 'req-d5c911b3-a370-4d88-862b-130a3e6067e6'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:52.015691 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-d5c911b3-a370-4d88-862b-130a3e6067e6 admin admin] The token cache is being invalidate because project f84f3c42dcc54722b46789217a4efd66 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81625) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:12:52.016057 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-d5c911b3-a370-4d88-862b-130a3e6067e6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:52.016759 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-d5c911b3-a370-4d88-862b-130a3e6067e6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:52.018324 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 998/1969] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:12:51 2022] DELETE /identity/v3/projects/f84f3c42dcc54722b46789217a4efd66 => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:12:52.022448 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a78f171a-b14d-4785-b56c-4b0cb920d0f1 service neutron] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:52.028623 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:52.032717 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a78f171a-b14d-4785-b56c-4b0cb920d0f1 service neutron] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:52.089460 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d8bda86-46d9-4496-8c54-ab3b424cb975 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:52.090000 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d8bda86-46d9-4496-8c54-ab3b424cb975 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:52.090279 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d8bda86-46d9-4496-8c54-ab3b424cb975 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:52.090549 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d8bda86-46d9-4496-8c54-ab3b424cb975 admin admin] PATH_INFO: `/v3/users/ce7cb64caf0843bbad8b508f4b2acd9d` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:52.091779 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d8bda86-46d9-4496-8c54-ab3b424cb975 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ce7cb64caf0843bbad8b508f4b2acd9d)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:52.093765 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d8bda86-46d9-4496-8c54-ab3b424cb975 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:52.140292 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-3d8bda86-46d9-4496-8c54-ab3b424cb975 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ce7cb64caf0843bbad8b508f4b2acd9d'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:52.140934 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-3d8bda86-46d9-4496-8c54-ab3b424cb975 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ce7cb64caf0843bbad8b508f4b2acd9d'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:52.141529 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-3d8bda86-46d9-4496-8c54-ab3b424cb975 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ce7cb64caf0843bbad8b508f4b2acd9d'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:52.153610 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-3d8bda86-46d9-4496-8c54-ab3b424cb975 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ce7cb64caf0843bbad8b508f4b2acd9d'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:52.160823 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-3d8bda86-46d9-4496-8c54-ab3b424cb975 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ce7cb64caf0843bbad8b508f4b2acd9d'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:52.173514 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 972/1970] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:12:52 2022] DELETE /identity/v3/users/ce7cb64caf0843bbad8b508f4b2acd9d => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:12:52.528714 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5c911b3-a370-4d88-862b-130a3e6067e6 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:52.530173 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-d5c911b3-a370-4d88-862b-130a3e6067e6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:52.546032 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5c911b3-a370-4d88-862b-130a3e6067e6 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:52.564944 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:DELETE=14,keystone:SELECT=84,keystone:INSERT=2 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:52.605319 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86689645-55d2-403f-87e3-dfc99c9e3441 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:52.605861 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86689645-55d2-403f-87e3-dfc99c9e3441 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:52.606134 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86689645-55d2-403f-87e3-dfc99c9e3441 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:52.606404 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86689645-55d2-403f-87e3-dfc99c9e3441 admin admin] PATH_INFO: `/v3/projects/a626219ad6c848509fe576bd569ae7c6` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:52.607524 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86689645-55d2-403f-87e3-dfc99c9e3441 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a626219ad6c848509fe576bd569ae7c6)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:52.609413 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86689645-55d2-403f-87e3-dfc99c9e3441 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:52.654505 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-86689645-55d2-403f-87e3-dfc99c9e3441 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a626219ad6c848509fe576bd569ae7c6', 'request_id': 'req-86689645-55d2-403f-87e3-dfc99c9e3441'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:52.661992 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-86689645-55d2-403f-87e3-dfc99c9e3441 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a626219ad6c848509fe576bd569ae7c6', 'request_id': 'req-86689645-55d2-403f-87e3-dfc99c9e3441'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:52.668175 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-86689645-55d2-403f-87e3-dfc99c9e3441 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a626219ad6c848509fe576bd569ae7c6', 'request_id': 'req-86689645-55d2-403f-87e3-dfc99c9e3441'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:52.678315 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-86689645-55d2-403f-87e3-dfc99c9e3441 admin admin] The token cache is being invalidate because project a626219ad6c848509fe576bd569ae7c6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81625) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:12:52.678596 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-86689645-55d2-403f-87e3-dfc99c9e3441 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:52.679233 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-86689645-55d2-403f-87e3-dfc99c9e3441 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:52.680683 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 999/1971] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:12:52 2022] DELETE /identity/v3/projects/a626219ad6c848509fe576bd569ae7c6 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 05:12:52.686449 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a57cf860-98be-4abc-a5d7-8a1d7675a081 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:52.686736 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a57cf860-98be-4abc-a5d7-8a1d7675a081 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:52.687071 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a57cf860-98be-4abc-a5d7-8a1d7675a081 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:52.706699 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-a57cf860-98be-4abc-a5d7-8a1d7675a081 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:12:52.708032 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-a57cf860-98be-4abc-a5d7-8a1d7675a081 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:52.766878 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 973/1972] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:12:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:12:52.777237 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81ba7399-6a05-4e39-8218-efdfb2becfec None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:52.777520 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81ba7399-6a05-4e39-8218-efdfb2becfec None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:52.777793 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81ba7399-6a05-4e39-8218-efdfb2becfec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:52.797993 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-81ba7399-6a05-4e39-8218-efdfb2becfec None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:12:52.799278 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-81ba7399-6a05-4e39-8218-efdfb2becfec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:52.807838 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1000/1973] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:12:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:12:52.811899 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a57cf860-98be-4abc-a5d7-8a1d7675a081 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:52.821900 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a57cf860-98be-4abc-a5d7-8a1d7675a081 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:52.826489 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6a5faf1-fd4c-4ec6-affc-8e022a207bce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:52.827186 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6a5faf1-fd4c-4ec6-affc-8e022a207bce admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:52.827461 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6a5faf1-fd4c-4ec6-affc-8e022a207bce admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:52.827728 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6a5faf1-fd4c-4ec6-affc-8e022a207bce admin admin] PATH_INFO: `/v3/domains` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:52.828361 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6a5faf1-fd4c-4ec6-affc-8e022a207bce admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:12:52.828658 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6a5faf1-fd4c-4ec6-affc-8e022a207bce admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:52.830215 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6a5faf1-fd4c-4ec6-affc-8e022a207bce admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:52.849377 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 974/1974] 10.222.0.77 () {58 vars in 1153 bytes} [Tue Aug 30 05:12:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:12:52.850531 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81ba7399-6a05-4e39-8218-efdfb2becfec None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:52.859983 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81ba7399-6a05-4e39-8218-efdfb2becfec None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:52.865215 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29ff2212-cd71-4744-9049-d6fe741cee23 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:52.865659 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ff2212-cd71-4744-9049-d6fe741cee23 admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:52.866072 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ff2212-cd71-4744-9049-d6fe741cee23 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:52.866365 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ff2212-cd71-4744-9049-d6fe741cee23 admin admin] PATH_INFO: `/v3/projects` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:52.867064 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29ff2212-cd71-4744-9049-d6fe741cee23 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:52.869134 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29ff2212-cd71-4744-9049-d6fe741cee23 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:52.900245 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1001/1975] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:12:52 2022] POST /identity/v3/projects => generated 385 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:12:52.904728 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6a5faf1-fd4c-4ec6-affc-8e022a207bce admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:52.915854 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6a5faf1-fd4c-4ec6-affc-8e022a207bce admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:52.927019 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:53.006014 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c5d9f3b-f5be-44a8-a8bb-b979f25b0fb6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:53.006014 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c5d9f3b-f5be-44a8-a8bb-b979f25b0fb6 admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:53.006014 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c5d9f3b-f5be-44a8-a8bb-b979f25b0fb6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:53.006014 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c5d9f3b-f5be-44a8-a8bb-b979f25b0fb6 admin admin] PATH_INFO: `/v3/users` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:53.007043 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c5d9f3b-f5be-44a8-a8bb-b979f25b0fb6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:53.009629 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c5d9f3b-f5be-44a8-a8bb-b979f25b0fb6 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:53.037984 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 975/1976] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:12:52 2022] POST /identity/v3/users => generated 293 bytes in 133 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:12:53.043815 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29ff2212-cd71-4744-9049-d6fe741cee23 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:53.056529 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29ff2212-cd71-4744-9049-d6fe741cee23 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:53.062579 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40085e4d-d5f0-4430-9547-92ea7d9332d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:53.063225 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40085e4d-d5f0-4430-9547-92ea7d9332d6 admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:53.063503 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40085e4d-d5f0-4430-9547-92ea7d9332d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:53.063814 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40085e4d-d5f0-4430-9547-92ea7d9332d6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:53.064419 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40085e4d-d5f0-4430-9547-92ea7d9332d6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:53.067172 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40085e4d-d5f0-4430-9547-92ea7d9332d6 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:53.082900 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1002/1977] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:12:53 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:12:53.085234 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c5d9f3b-f5be-44a8-a8bb-b979f25b0fb6 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:53.095700 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c5d9f3b-f5be-44a8-a8bb-b979f25b0fb6 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:53.101275 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62e5e745-fd25-44b9-b624-573b34dc0b7c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:53.102121 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e5e745-fd25-44b9-b624-573b34dc0b7c admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:53.102463 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e5e745-fd25-44b9-b624-573b34dc0b7c admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:53.102802 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e5e745-fd25-44b9-b624-573b34dc0b7c admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:53.103579 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62e5e745-fd25-44b9-b624-573b34dc0b7c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:53.105433 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62e5e745-fd25-44b9-b624-573b34dc0b7c admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:53.117376 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 976/1978] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:12:53 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 05:12:53.122417 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40085e4d-d5f0-4430-9547-92ea7d9332d6 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:53.133036 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40085e4d-d5f0-4430-9547-92ea7d9332d6 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:53.137574 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9245a98-8a9f-435b-a6f3-749c68c00dfc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:53.138243 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9245a98-8a9f-435b-a6f3-749c68c00dfc admin admin] REQUEST_METHOD: `PUT` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:53.138515 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9245a98-8a9f-435b-a6f3-749c68c00dfc admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:53.138769 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9245a98-8a9f-435b-a6f3-749c68c00dfc admin admin] PATH_INFO: `/v3/projects/1a8ca4ec40e84dbfa8adf3b801e31baf/users/8589cbfccb2d4b0b9e294bcf4583edb3/roles/6f03d86980c34c36b44a8d6764ff4fda` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:53.156055 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9245a98-8a9f-435b-a6f3-749c68c00dfc admin admin] RBAC: Authorizing `identity:create_grant(project_id=1a8ca4ec40e84dbfa8adf3b801e31baf, user_id=8589cbfccb2d4b0b9e294bcf4583edb3, role_id=6f03d86980c34c36b44a8d6764ff4fda)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:53.158762 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9245a98-8a9f-435b-a6f3-749c68c00dfc admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:53.171241 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1003/1979] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:12:53 2022] PUT /identity/v3/projects/1a8ca4ec40e84dbfa8adf3b801e31baf/users/8589cbfccb2d4b0b9e294bcf4583edb3/roles/6f03d86980c34c36b44a8d6764ff4fda => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:12:53.182485 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-533bfcea-07cb-4688-8633-fe64f3b7d44f None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:53.182844 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-533bfcea-07cb-4688-8633-fe64f3b7d44f None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:53.183115 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-533bfcea-07cb-4688-8633-fe64f3b7d44f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:53.219655 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-533bfcea-07cb-4688-8633-fe64f3b7d44f None None] MFA Rules not processed for user `8589cbfccb2d4b0b9e294bcf4583edb3`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:12:53.221186 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-533bfcea-07cb-4688-8633-fe64f3b7d44f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:53.290903 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 977/1980] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:12:53 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:12:53.295391 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9245a98-8a9f-435b-a6f3-749c68c00dfc admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:53.347752 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:53.362957 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9245a98-8a9f-435b-a6f3-749c68c00dfc admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:53.367742 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d1e3ab8-a7f8-4b29-a4b7-ca4016d06cbe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:53.368433 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d1e3ab8-a7f8-4b29-a4b7-ca4016d06cbe admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:53.368741 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d1e3ab8-a7f8-4b29-a4b7-ca4016d06cbe admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:53.369125 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d1e3ab8-a7f8-4b29-a4b7-ca4016d06cbe admin admin] PATH_INFO: `/v3/projects` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:53.369877 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d1e3ab8-a7f8-4b29-a4b7-ca4016d06cbe admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:53.372014 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d1e3ab8-a7f8-4b29-a4b7-ca4016d06cbe admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:53.406634 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1004/1981] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:12:53 2022] POST /identity/v3/projects => generated 383 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:12:53.410733 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-533bfcea-07cb-4688-8633-fe64f3b7d44f None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:53.422306 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-533bfcea-07cb-4688-8633-fe64f3b7d44f None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:53.497308 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c89236e-66a7-4902-b2f0-2362d5e14348 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:53.498531 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c89236e-66a7-4902-b2f0-2362d5e14348 admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:53.498839 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c89236e-66a7-4902-b2f0-2362d5e14348 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:53.499514 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c89236e-66a7-4902-b2f0-2362d5e14348 admin admin] PATH_INFO: `/v3/users` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:53.500032 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c89236e-66a7-4902-b2f0-2362d5e14348 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:53.507337 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c89236e-66a7-4902-b2f0-2362d5e14348 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:53.530734 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 978/1982] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:12:53 2022] POST /identity/v3/users => generated 292 bytes in 120 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 05:12:53.535103 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d1e3ab8-a7f8-4b29-a4b7-ca4016d06cbe admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:53.546616 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d1e3ab8-a7f8-4b29-a4b7-ca4016d06cbe admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:53.552436 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1462615-5f90-48fe-a7e8-e8abf087ddbf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:53.553142 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1462615-5f90-48fe-a7e8-e8abf087ddbf admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:53.553494 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1462615-5f90-48fe-a7e8-e8abf087ddbf admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:53.553825 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1462615-5f90-48fe-a7e8-e8abf087ddbf admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:53.554568 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1462615-5f90-48fe-a7e8-e8abf087ddbf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:53.556332 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1462615-5f90-48fe-a7e8-e8abf087ddbf admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:53.571532 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1005/1983] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:12:53 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 05:12:53.575181 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c89236e-66a7-4902-b2f0-2362d5e14348 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:53.585460 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c89236e-66a7-4902-b2f0-2362d5e14348 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:53.590926 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-634ba5da-761d-430b-89d5-ee9fdc2c7403 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:53.591525 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-634ba5da-761d-430b-89d5-ee9fdc2c7403 admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:53.591799 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-634ba5da-761d-430b-89d5-ee9fdc2c7403 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:53.592063 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-634ba5da-761d-430b-89d5-ee9fdc2c7403 admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:53.592700 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-634ba5da-761d-430b-89d5-ee9fdc2c7403 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:53.594744 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-634ba5da-761d-430b-89d5-ee9fdc2c7403 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:53.597600 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:53.611084 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 979/1984] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:12:53 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 05:12:53.612887 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1462615-5f90-48fe-a7e8-e8abf087ddbf admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:53.624064 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1462615-5f90-48fe-a7e8-e8abf087ddbf admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:53.629756 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-171c72e8-46ad-4489-a69e-e235237eacad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:53.630401 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-171c72e8-46ad-4489-a69e-e235237eacad admin admin] REQUEST_METHOD: `PUT` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:53.630702 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-171c72e8-46ad-4489-a69e-e235237eacad admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:53.631038 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-171c72e8-46ad-4489-a69e-e235237eacad admin admin] PATH_INFO: `/v3/projects/fc64f952d2b14ead882fb7b39f11e304/users/fdd423e995984e709a7bbde68abfc986/roles/6f03d86980c34c36b44a8d6764ff4fda` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:53.645714 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-171c72e8-46ad-4489-a69e-e235237eacad admin admin] RBAC: Authorizing `identity:create_grant(project_id=fc64f952d2b14ead882fb7b39f11e304, user_id=fdd423e995984e709a7bbde68abfc986, role_id=6f03d86980c34c36b44a8d6764ff4fda)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:53.648418 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-171c72e8-46ad-4489-a69e-e235237eacad admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:53.661454 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1006/1985] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:12:53 2022] PUT /identity/v3/projects/fc64f952d2b14ead882fb7b39f11e304/users/fdd423e995984e709a7bbde68abfc986/roles/6f03d86980c34c36b44a8d6764ff4fda => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:12:53.671733 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76e08111-a245-48b3-aa55-1342b0d9c64d None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:53.672016 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76e08111-a245-48b3-aa55-1342b0d9c64d None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:53.672285 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76e08111-a245-48b3-aa55-1342b0d9c64d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:53.700782 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-76e08111-a245-48b3-aa55-1342b0d9c64d None None] MFA Rules not processed for user `fdd423e995984e709a7bbde68abfc986`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:12:53.701991 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-76e08111-a245-48b3-aa55-1342b0d9c64d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:53.765711 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 980/1986] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:12:53 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:12:53.770694 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-171c72e8-46ad-4489-a69e-e235237eacad admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:53.831449 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-171c72e8-46ad-4489-a69e-e235237eacad admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:53.836064 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c70a4015-1fcd-43b6-b5e2-2531e403fcba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:53.836652 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c70a4015-1fcd-43b6-b5e2-2531e403fcba admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:53.836934 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c70a4015-1fcd-43b6-b5e2-2531e403fcba admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:53.837203 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c70a4015-1fcd-43b6-b5e2-2531e403fcba admin admin] PATH_INFO: `/v3/projects` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:53.837818 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c70a4015-1fcd-43b6-b5e2-2531e403fcba admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:53.839737 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c70a4015-1fcd-43b6-b5e2-2531e403fcba admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:53.865454 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1007/1987] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:12:53 2022] POST /identity/v3/projects => generated 385 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:12:53.869517 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76e08111-a245-48b3-aa55-1342b0d9c64d None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:53.880584 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76e08111-a245-48b3-aa55-1342b0d9c64d None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:53.942500 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1585e07-0f2e-4a8d-b7a2-1894ddc8c767 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:53.943128 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1585e07-0f2e-4a8d-b7a2-1894ddc8c767 admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:53.943410 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1585e07-0f2e-4a8d-b7a2-1894ddc8c767 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:53.943682 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1585e07-0f2e-4a8d-b7a2-1894ddc8c767 admin admin] PATH_INFO: `/v3/users` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:53.944313 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1585e07-0f2e-4a8d-b7a2-1894ddc8c767 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:53.946233 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1585e07-0f2e-4a8d-b7a2-1894ddc8c767 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:53.967128 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 981/1988] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:12:53 2022] POST /identity/v3/users => generated 299 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:12:53.971751 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c70a4015-1fcd-43b6-b5e2-2531e403fcba admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:53.989077 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c70a4015-1fcd-43b6-b5e2-2531e403fcba admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:53.993798 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc7024fa-0178-4040-b5d6-287fac899970 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:53.994392 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc7024fa-0178-4040-b5d6-287fac899970 admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:53.994693 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc7024fa-0178-4040-b5d6-287fac899970 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:53.994999 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc7024fa-0178-4040-b5d6-287fac899970 admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:53.995633 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc7024fa-0178-4040-b5d6-287fac899970 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:53.997212 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc7024fa-0178-4040-b5d6-287fac899970 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:54.008457 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1008/1989] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:12:53 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:12:54.012796 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1585e07-0f2e-4a8d-b7a2-1894ddc8c767 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:54.023675 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1585e07-0f2e-4a8d-b7a2-1894ddc8c767 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:54.029024 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f14b5957-e7cc-4a4f-9553-d64502b4894b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:54.029976 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f14b5957-e7cc-4a4f-9553-d64502b4894b admin admin] REQUEST_METHOD: `PUT` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:54.030099 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f14b5957-e7cc-4a4f-9553-d64502b4894b admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:54.030426 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f14b5957-e7cc-4a4f-9553-d64502b4894b admin admin] PATH_INFO: `/v3/projects/91ff10a8a8a44f89917cf2af46319564/users/00713a8ed26e43958ff9012cc9afe2eb/roles/9d9d5d2c270146488249b7ed943280db` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:54.047445 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f14b5957-e7cc-4a4f-9553-d64502b4894b admin admin] RBAC: Authorizing `identity:create_grant(project_id=91ff10a8a8a44f89917cf2af46319564, user_id=00713a8ed26e43958ff9012cc9afe2eb, role_id=9d9d5d2c270146488249b7ed943280db)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:54.050295 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f14b5957-e7cc-4a4f-9553-d64502b4894b admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:54.065760 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 982/1990] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:12:54 2022] PUT /identity/v3/projects/91ff10a8a8a44f89917cf2af46319564/users/00713a8ed26e43958ff9012cc9afe2eb/roles/9d9d5d2c270146488249b7ed943280db => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:12:54.070061 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc7024fa-0178-4040-b5d6-287fac899970 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:54.080445 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc7024fa-0178-4040-b5d6-287fac899970 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:54.176651 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6039428e-6807-41f5-af87-e15467c5b954 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:54.176935 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6039428e-6807-41f5-af87-e15467c5b954 admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:54.177288 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6039428e-6807-41f5-af87-e15467c5b954 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:54.177628 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6039428e-6807-41f5-af87-e15467c5b954 admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:54.178339 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6039428e-6807-41f5-af87-e15467c5b954 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:54.180039 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6039428e-6807-41f5-af87-e15467c5b954 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:54.192030 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1009/1991] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:12:54 2022] GET /identity/v3/roles => generated 1420 bytes in 123 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:12:54.196308 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f14b5957-e7cc-4a4f-9553-d64502b4894b admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:54.206993 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f14b5957-e7cc-4a4f-9553-d64502b4894b admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:54.213529 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b430eb6-a0b4-4aac-8f96-5246b9abe17e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:54.214197 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b430eb6-a0b4-4aac-8f96-5246b9abe17e admin admin] REQUEST_METHOD: `PUT` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:54.214472 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b430eb6-a0b4-4aac-8f96-5246b9abe17e admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:54.214743 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b430eb6-a0b4-4aac-8f96-5246b9abe17e admin admin] PATH_INFO: `/v3/projects/91ff10a8a8a44f89917cf2af46319564/users/00713a8ed26e43958ff9012cc9afe2eb/roles/9d9d5d2c270146488249b7ed943280db` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:54.217441 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b430eb6-a0b4-4aac-8f96-5246b9abe17e admin admin] RBAC: Authorizing `identity:create_grant(project_id=91ff10a8a8a44f89917cf2af46319564, user_id=00713a8ed26e43958ff9012cc9afe2eb, role_id=9d9d5d2c270146488249b7ed943280db)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:54.226446 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b430eb6-a0b4-4aac-8f96-5246b9abe17e admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:54.249671 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 983/1992] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:12:54 2022] PUT /identity/v3/projects/91ff10a8a8a44f89917cf2af46319564/users/00713a8ed26e43958ff9012cc9afe2eb/roles/9d9d5d2c270146488249b7ed943280db => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:12:54.262172 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cd50f59-665a-49b8-86c5-c8e69c6ce1c7 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:54.262536 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cd50f59-665a-49b8-86c5-c8e69c6ce1c7 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:54.262837 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cd50f59-665a-49b8-86c5-c8e69c6ce1c7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:54.285470 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:54.299068 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-1cd50f59-665a-49b8-86c5-c8e69c6ce1c7 None None] MFA Rules not processed for user `00713a8ed26e43958ff9012cc9afe2eb`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:12:54.301272 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-1cd50f59-665a-49b8-86c5-c8e69c6ce1c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:54.397971 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1010/1993] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:12:54 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:12:54.409947 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b430eb6-a0b4-4aac-8f96-5246b9abe17e admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:54.414864 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-5b430eb6-a0b4-4aac-8f96-5246b9abe17e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:54.473449 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:54.499657 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b430eb6-a0b4-4aac-8f96-5246b9abe17e admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:54.503278 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dc7d520-1096-41a0-b958-3d9b13980144 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '019a79b2affe478b8a30d1c538d0415c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:54.503995 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dc7d520-1096-41a0-b958-3d9b13980144 service cinder] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:54.504277 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dc7d520-1096-41a0-b958-3d9b13980144 service cinder] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:54.504556 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dc7d520-1096-41a0-b958-3d9b13980144 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:54.567400 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dc7d520-1096-41a0-b958-3d9b13980144 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:54.568867 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dc7d520-1096-41a0-b958-3d9b13980144 service cinder] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:54.578650 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 984/1994] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:12:54 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 169 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:12:55.374708 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cd50f59-665a-49b8-86c5-c8e69c6ce1c7 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:55.376224 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-1cd50f59-665a-49b8-86c5-c8e69c6ce1c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:55.465278 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cd50f59-665a-49b8-86c5-c8e69c6ce1c7 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:55.470675 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed15f97-7628-4e7a-940d-12c42e6fb956 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:55.472095 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed15f97-7628-4e7a-940d-12c42e6fb956 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:55.472440 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed15f97-7628-4e7a-940d-12c42e6fb956 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:55.472777 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed15f97-7628-4e7a-940d-12c42e6fb956 admin admin] PATH_INFO: `/v3/users/f3358a3a394f48838bd35aeed77b5d88` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:55.474083 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bed15f97-7628-4e7a-940d-12c42e6fb956 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f3358a3a394f48838bd35aeed77b5d88)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:55.476154 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bed15f97-7628-4e7a-940d-12c42e6fb956 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:55.554261 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-bed15f97-7628-4e7a-940d-12c42e6fb956 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f3358a3a394f48838bd35aeed77b5d88'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:55.562243 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dc7d520-1096-41a0-b958-3d9b13980144 service cinder] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:55.563817 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-1dc7d520-1096-41a0-b958-3d9b13980144 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:55.564721 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-bed15f97-7628-4e7a-940d-12c42e6fb956 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f3358a3a394f48838bd35aeed77b5d88'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:55.582440 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-bed15f97-7628-4e7a-940d-12c42e6fb956 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f3358a3a394f48838bd35aeed77b5d88'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:55.584196 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dc7d520-1096-41a0-b958-3d9b13980144 service cinder] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:55.592720 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-424738f3-2c8b-491c-b239-79c95af95568 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:55.593530 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-424738f3-2c8b-491c-b239-79c95af95568 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:55.593865 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-424738f3-2c8b-491c-b239-79c95af95568 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:55.594515 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-424738f3-2c8b-491c-b239-79c95af95568 admin admin] PATH_INFO: `/v3/users/8795355dcaa14d30954ef3739d32e1dc` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:55.594748 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-bed15f97-7628-4e7a-940d-12c42e6fb956 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f3358a3a394f48838bd35aeed77b5d88'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:55.595805 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-bed15f97-7628-4e7a-940d-12c42e6fb956 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f3358a3a394f48838bd35aeed77b5d88'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:55.596886 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-424738f3-2c8b-491c-b239-79c95af95568 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8795355dcaa14d30954ef3739d32e1dc)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:55.599099 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-424738f3-2c8b-491c-b239-79c95af95568 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:55.604650 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1011/1995] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:12:55 2022] DELETE /identity/v3/users/f3358a3a394f48838bd35aeed77b5d88 => generated 0 bytes in 230 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:12:55.609343 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a49c28-7915-4da8-88bc-35cfa98fb4db None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:55.609951 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a49c28-7915-4da8-88bc-35cfa98fb4db None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:55.610408 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a49c28-7915-4da8-88bc-35cfa98fb4db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:55.633577 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-10a49c28-7915-4da8-88bc-35cfa98fb4db None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:12:55.634999 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-10a49c28-7915-4da8-88bc-35cfa98fb4db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:55.647177 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:55.658578 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-424738f3-2c8b-491c-b239-79c95af95568 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8795355dcaa14d30954ef3739d32e1dc'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:55.660164 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-424738f3-2c8b-491c-b239-79c95af95568 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8795355dcaa14d30954ef3739d32e1dc'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:55.660905 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-424738f3-2c8b-491c-b239-79c95af95568 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8795355dcaa14d30954ef3739d32e1dc'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:55.680594 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-424738f3-2c8b-491c-b239-79c95af95568 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8795355dcaa14d30954ef3739d32e1dc'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:55.690563 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-424738f3-2c8b-491c-b239-79c95af95568 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8795355dcaa14d30954ef3739d32e1dc'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:55.706898 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 985/1996] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:12:55 2022] DELETE /identity/v3/users/8795355dcaa14d30954ef3739d32e1dc => generated 0 bytes in 145 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 05:12:55.711537 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f931aaf-fcc5-4ebe-a4ea-c35626b24af3 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:55.711828 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f931aaf-fcc5-4ebe-a4ea-c35626b24af3 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:55.712193 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f931aaf-fcc5-4ebe-a4ea-c35626b24af3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:55.713004 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1012/1997] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:12:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:12:55.725527 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10a49c28-7915-4da8-88bc-35cfa98fb4db None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:55.727165 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-10a49c28-7915-4da8-88bc-35cfa98fb4db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:55.732715 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-5f931aaf-fcc5-4ebe-a4ea-c35626b24af3 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:12:55.734000 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-5f931aaf-fcc5-4ebe-a4ea-c35626b24af3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:55.792906 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10a49c28-7915-4da8-88bc-35cfa98fb4db None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:55.795694 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 986/1998] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:12:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 05:12:55.805153 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f931aaf-fcc5-4ebe-a4ea-c35626b24af3 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:55.807028 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-5f931aaf-fcc5-4ebe-a4ea-c35626b24af3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:55.818228 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:55.851104 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-905b2e7a-6e6d-433a-9200-74f5f34fc77a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:55.851720 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-905b2e7a-6e6d-433a-9200-74f5f34fc77a service neutron] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:55.852003 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-905b2e7a-6e6d-433a-9200-74f5f34fc77a service neutron] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:55.852276 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-905b2e7a-6e6d-433a-9200-74f5f34fc77a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:55.872609 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-905b2e7a-6e6d-433a-9200-74f5f34fc77a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:55.872609 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-905b2e7a-6e6d-433a-9200-74f5f34fc77a service neutron] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:55.882958 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f931aaf-fcc5-4ebe-a4ea-c35626b24af3 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:55.883137 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1013/1999] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:12:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 158 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:12:55.887378 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77a5166f-70e7-44a4-b49e-16f342e3e2d4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:55.887999 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77a5166f-70e7-44a4-b49e-16f342e3e2d4 service neutron] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:55.888279 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77a5166f-70e7-44a4-b49e-16f342e3e2d4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:55.888557 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77a5166f-70e7-44a4-b49e-16f342e3e2d4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:55.899424 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77a5166f-70e7-44a4-b49e-16f342e3e2d4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:55.900921 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77a5166f-70e7-44a4-b49e-16f342e3e2d4 service neutron] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:55.911569 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 987/2000] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:12:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 05:12:56.380754 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-905b2e7a-6e6d-433a-9200-74f5f34fc77a service neutron] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:56.382896 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-905b2e7a-6e6d-433a-9200-74f5f34fc77a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:56.402518 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-905b2e7a-6e6d-433a-9200-74f5f34fc77a service neutron] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:56.418573 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43417abc-9f5d-4a02-8700-62ef2a97cd1d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:56.419490 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43417abc-9f5d-4a02-8700-62ef2a97cd1d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:56.420002 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43417abc-9f5d-4a02-8700-62ef2a97cd1d admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:56.425450 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43417abc-9f5d-4a02-8700-62ef2a97cd1d admin admin] PATH_INFO: `/v3/projects/d6884877d69d43b4a31ff6a2bee7f747` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:56.427058 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43417abc-9f5d-4a02-8700-62ef2a97cd1d admin admin] RBAC: Authorizing `identity:delete_project(project_id=d6884877d69d43b4a31ff6a2bee7f747)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:56.429488 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43417abc-9f5d-4a02-8700-62ef2a97cd1d admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:56.485109 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77a5166f-70e7-44a4-b49e-16f342e3e2d4 service neutron] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:56.487433 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-77a5166f-70e7-44a4-b49e-16f342e3e2d4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:56.509788 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77a5166f-70e7-44a4-b49e-16f342e3e2d4 service neutron] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:56.517642 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-43417abc-9f5d-4a02-8700-62ef2a97cd1d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd6884877d69d43b4a31ff6a2bee7f747', 'request_id': 'req-43417abc-9f5d-4a02-8700-62ef2a97cd1d'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:56.550293 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-43417abc-9f5d-4a02-8700-62ef2a97cd1d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd6884877d69d43b4a31ff6a2bee7f747', 'request_id': 'req-43417abc-9f5d-4a02-8700-62ef2a97cd1d'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:56.559967 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-43417abc-9f5d-4a02-8700-62ef2a97cd1d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd6884877d69d43b4a31ff6a2bee7f747', 'request_id': 'req-43417abc-9f5d-4a02-8700-62ef2a97cd1d'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:56.567956 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:56.578972 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-43417abc-9f5d-4a02-8700-62ef2a97cd1d admin admin] The token cache is being invalidate because project d6884877d69d43b4a31ff6a2bee7f747 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81625) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:12:56.579510 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-43417abc-9f5d-4a02-8700-62ef2a97cd1d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:56.580880 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-43417abc-9f5d-4a02-8700-62ef2a97cd1d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:56.583853 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1014/2001] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:12:56 2022] DELETE /identity/v3/projects/d6884877d69d43b4a31ff6a2bee7f747 => generated 0 bytes in 204 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:12:56.586419 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43417abc-9f5d-4a02-8700-62ef2a97cd1d admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:56.588037 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-43417abc-9f5d-4a02-8700-62ef2a97cd1d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:56.608640 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52727c93-cf09-4a17-99c7-315a577c9bd1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:56.609285 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52727c93-cf09-4a17-99c7-315a577c9bd1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:56.609565 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52727c93-cf09-4a17-99c7-315a577c9bd1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:56.609838 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52727c93-cf09-4a17-99c7-315a577c9bd1 admin admin] PATH_INFO: `/v3/projects/7884824e60d248b683b215d10aa3fadf` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:56.611198 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52727c93-cf09-4a17-99c7-315a577c9bd1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7884824e60d248b683b215d10aa3fadf)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:56.613815 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52727c93-cf09-4a17-99c7-315a577c9bd1 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:56.662996 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43417abc-9f5d-4a02-8700-62ef2a97cd1d admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:56.679389 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-52727c93-cf09-4a17-99c7-315a577c9bd1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7884824e60d248b683b215d10aa3fadf', 'request_id': 'req-52727c93-cf09-4a17-99c7-315a577c9bd1'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:56.691719 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-52727c93-cf09-4a17-99c7-315a577c9bd1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7884824e60d248b683b215d10aa3fadf', 'request_id': 'req-52727c93-cf09-4a17-99c7-315a577c9bd1'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:56.699670 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-52727c93-cf09-4a17-99c7-315a577c9bd1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7884824e60d248b683b215d10aa3fadf', 'request_id': 'req-52727c93-cf09-4a17-99c7-315a577c9bd1'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:56.718885 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-52727c93-cf09-4a17-99c7-315a577c9bd1 admin admin] The token cache is being invalidate because project 7884824e60d248b683b215d10aa3fadf was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81624) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:12:56.719468 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-52727c93-cf09-4a17-99c7-315a577c9bd1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:56.720103 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-52727c93-cf09-4a17-99c7-315a577c9bd1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:56.722310 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 988/2002] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:12:56 2022] DELETE /identity/v3/projects/7884824e60d248b683b215d10aa3fadf => generated 0 bytes in 238 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 05:12:56.723116 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52727c93-cf09-4a17-99c7-315a577c9bd1 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:56.725516 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-52727c93-cf09-4a17-99c7-315a577c9bd1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:56.758666 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7944ea74-af95-4743-929e-8031e0d79ee0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '019a79b2affe478b8a30d1c538d0415c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:56.759774 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7944ea74-af95-4743-929e-8031e0d79ee0 service cinder] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:56.760064 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7944ea74-af95-4743-929e-8031e0d79ee0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:56.760345 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7944ea74-af95-4743-929e-8031e0d79ee0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:56.767767 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-7944ea74-af95-4743-929e-8031e0d79ee0 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:56.805981 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:56.834376 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52727c93-cf09-4a17-99c7-315a577c9bd1 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:56.839257 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a09b130a-975b-4208-8beb-4bda295923f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:56.839905 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a09b130a-975b-4208-8beb-4bda295923f8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:56.840188 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a09b130a-975b-4208-8beb-4bda295923f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:56.840464 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a09b130a-975b-4208-8beb-4bda295923f8 admin admin] PATH_INFO: `/v3/users/425ed8189fc247b4a72dd6b1ef8bf85c` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:56.841629 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a09b130a-975b-4208-8beb-4bda295923f8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=425ed8189fc247b4a72dd6b1ef8bf85c)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:56.843608 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a09b130a-975b-4208-8beb-4bda295923f8 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:56.907912 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7944ea74-af95-4743-929e-8031e0d79ee0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:56.910447 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7944ea74-af95-4743-929e-8031e0d79ee0 service cinder] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:56.911684 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-a09b130a-975b-4208-8beb-4bda295923f8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '425ed8189fc247b4a72dd6b1ef8bf85c'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:56.912457 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-a09b130a-975b-4208-8beb-4bda295923f8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '425ed8189fc247b4a72dd6b1ef8bf85c'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:56.913603 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-a09b130a-975b-4208-8beb-4bda295923f8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '425ed8189fc247b4a72dd6b1ef8bf85c'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:56.927832 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1015/2003] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:12:56 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 343 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:12:56.928851 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7944ea74-af95-4743-929e-8031e0d79ee0 service cinder] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:56.934506 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-7944ea74-af95-4743-929e-8031e0d79ee0 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:56.947713 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-a09b130a-975b-4208-8beb-4bda295923f8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '425ed8189fc247b4a72dd6b1ef8bf85c'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:56.962412 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-a09b130a-975b-4208-8beb-4bda295923f8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '425ed8189fc247b4a72dd6b1ef8bf85c'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:56.976892 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7944ea74-af95-4743-929e-8031e0d79ee0 service cinder] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:56.988621 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dec4bfd1-635b-4452-aa3b-8aaabb6682f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:56.989381 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec4bfd1-635b-4452-aa3b-8aaabb6682f0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:56.989681 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec4bfd1-635b-4452-aa3b-8aaabb6682f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:56.989892 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec4bfd1-635b-4452-aa3b-8aaabb6682f0 admin admin] PATH_INFO: `/v3/users/7f94c03417f14d2dbdd70bf158b6c4d3` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:56.991405 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dec4bfd1-635b-4452-aa3b-8aaabb6682f0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7f94c03417f14d2dbdd70bf158b6c4d3)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:57.000787 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 989/2004] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:12:56 2022] DELETE /identity/v3/users/425ed8189fc247b4a72dd6b1ef8bf85c => generated 0 bytes in 278 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:12:57.006469 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dec4bfd1-635b-4452-aa3b-8aaabb6682f0 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:57.109860 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-dec4bfd1-635b-4452-aa3b-8aaabb6682f0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7f94c03417f14d2dbdd70bf158b6c4d3'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:57.125717 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-dec4bfd1-635b-4452-aa3b-8aaabb6682f0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7f94c03417f14d2dbdd70bf158b6c4d3'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:57.141824 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:57.176712 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-dec4bfd1-635b-4452-aa3b-8aaabb6682f0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7f94c03417f14d2dbdd70bf158b6c4d3'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:57.199364 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-dec4bfd1-635b-4452-aa3b-8aaabb6682f0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7f94c03417f14d2dbdd70bf158b6c4d3'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:57.201356 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-dec4bfd1-635b-4452-aa3b-8aaabb6682f0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7f94c03417f14d2dbdd70bf158b6c4d3'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:57.214997 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1016/2005] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:12:56 2022] DELETE /identity/v3/users/7f94c03417f14d2dbdd70bf158b6c4d3 => generated 0 bytes in 286 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:12:57.883172 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a09b130a-975b-4208-8beb-4bda295923f8 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:57.884659 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-a09b130a-975b-4208-8beb-4bda295923f8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:57.963602 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a09b130a-975b-4208-8beb-4bda295923f8 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:58.038278 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e07db297-6ffd-4168-abf2-e2038d13f950 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '019a79b2affe478b8a30d1c538d0415c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:58.039187 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e07db297-6ffd-4168-abf2-e2038d13f950 service cinder] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:58.039569 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e07db297-6ffd-4168-abf2-e2038d13f950 service cinder] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:58.040176 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e07db297-6ffd-4168-abf2-e2038d13f950 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:58.042855 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-e07db297-6ffd-4168-abf2-e2038d13f950 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:58.089101 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:58.144306 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e07db297-6ffd-4168-abf2-e2038d13f950 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:58.146041 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e07db297-6ffd-4168-abf2-e2038d13f950 service cinder] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:58.162677 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 990/2006] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:12:57 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 280 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:12:58.197778 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dec4bfd1-635b-4452-aa3b-8aaabb6682f0 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:58.199327 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-dec4bfd1-635b-4452-aa3b-8aaabb6682f0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:58.324040 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dec4bfd1-635b-4452-aa3b-8aaabb6682f0 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:58.347918 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-942ac022-2e9a-44d6-915c-78074b9739df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:58.348698 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-942ac022-2e9a-44d6-915c-78074b9739df admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:58.349050 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-942ac022-2e9a-44d6-915c-78074b9739df admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:58.349392 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-942ac022-2e9a-44d6-915c-78074b9739df admin admin] PATH_INFO: `/v3/projects/3079e3ce4f124d4591f17f78e271d29c` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:58.351064 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-942ac022-2e9a-44d6-915c-78074b9739df admin admin] RBAC: Authorizing `identity:delete_project(project_id=3079e3ce4f124d4591f17f78e271d29c)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:58.353258 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-942ac022-2e9a-44d6-915c-78074b9739df admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:58.426555 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-942ac022-2e9a-44d6-915c-78074b9739df admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3079e3ce4f124d4591f17f78e271d29c', 'request_id': 'req-942ac022-2e9a-44d6-915c-78074b9739df'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:58.438016 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-942ac022-2e9a-44d6-915c-78074b9739df admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3079e3ce4f124d4591f17f78e271d29c', 'request_id': 'req-942ac022-2e9a-44d6-915c-78074b9739df'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:58.449220 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-942ac022-2e9a-44d6-915c-78074b9739df admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3079e3ce4f124d4591f17f78e271d29c', 'request_id': 'req-942ac022-2e9a-44d6-915c-78074b9739df'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:58.466241 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-942ac022-2e9a-44d6-915c-78074b9739df admin admin] The token cache is being invalidate because project 3079e3ce4f124d4591f17f78e271d29c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81625) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:12:58.466860 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-942ac022-2e9a-44d6-915c-78074b9739df admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:58.467768 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-942ac022-2e9a-44d6-915c-78074b9739df admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:58.470863 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1017/2007] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:12:58 2022] DELETE /identity/v3/projects/3079e3ce4f124d4591f17f78e271d29c => generated 0 bytes in 273 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:12:58.475698 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65f5f923-1c52-4ba0-9647-29c278b46b0e None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:58.476217 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65f5f923-1c52-4ba0-9647-29c278b46b0e None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:58.476706 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65f5f923-1c52-4ba0-9647-29c278b46b0e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:58.499816 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-65f5f923-1c52-4ba0-9647-29c278b46b0e None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:12:58.501028 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-65f5f923-1c52-4ba0-9647-29c278b46b0e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:58.602045 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 991/2008] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:12:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 05:12:58.610075 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce9963b-8550-499e-9375-fb87421699d2 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:58.610507 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce9963b-8550-499e-9375-fb87421699d2 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:58.610935 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce9963b-8550-499e-9375-fb87421699d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:58.634186 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-fce9963b-8550-499e-9375-fb87421699d2 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:12:58.635659 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-fce9963b-8550-499e-9375-fb87421699d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:58.646855 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1018/2009] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:12:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:12:58.651106 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65f5f923-1c52-4ba0-9647-29c278b46b0e None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:58.663474 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65f5f923-1c52-4ba0-9647-29c278b46b0e None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:58.671259 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab7ca5e4-7c91-4f3c-811e-cd853a197865 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:58.671887 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab7ca5e4-7c91-4f3c-811e-cd853a197865 admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:58.672168 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab7ca5e4-7c91-4f3c-811e-cd853a197865 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:58.672511 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab7ca5e4-7c91-4f3c-811e-cd853a197865 admin admin] PATH_INFO: `/v3/domains` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:58.673590 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab7ca5e4-7c91-4f3c-811e-cd853a197865 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:12:58.673896 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab7ca5e4-7c91-4f3c-811e-cd853a197865 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:58.675843 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab7ca5e4-7c91-4f3c-811e-cd853a197865 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:58.690026 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 992/2010] 10.222.0.77 () {58 vars in 1153 bytes} [Tue Aug 30 05:12:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:12:58.694542 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fce9963b-8550-499e-9375-fb87421699d2 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:58.707472 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fce9963b-8550-499e-9375-fb87421699d2 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:58.715184 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-332d6b61-613e-4764-a6ee-5c3b46178201 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:58.715936 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-332d6b61-613e-4764-a6ee-5c3b46178201 admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:58.716280 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-332d6b61-613e-4764-a6ee-5c3b46178201 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:58.716617 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-332d6b61-613e-4764-a6ee-5c3b46178201 admin admin] PATH_INFO: `/v3/projects` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:58.718015 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-332d6b61-613e-4764-a6ee-5c3b46178201 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:58.720582 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-332d6b61-613e-4764-a6ee-5c3b46178201 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:58.772366 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1019/2011] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:12:58 2022] POST /identity/v3/projects => generated 381 bytes in 78 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 05:12:58.775316 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab7ca5e4-7c91-4f3c-811e-cd853a197865 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:58.788314 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab7ca5e4-7c91-4f3c-811e-cd853a197865 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:58.874070 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb297354-f73a-494d-b8df-0d7b977f4da6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:58.874696 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb297354-f73a-494d-b8df-0d7b977f4da6 admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:58.875060 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb297354-f73a-494d-b8df-0d7b977f4da6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:58.875275 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb297354-f73a-494d-b8df-0d7b977f4da6 admin admin] PATH_INFO: `/v3/users` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:58.875905 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb297354-f73a-494d-b8df-0d7b977f4da6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:58.890204 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb297354-f73a-494d-b8df-0d7b977f4da6 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:58.918439 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 993/2012] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:12:58 2022] POST /identity/v3/users => generated 291 bytes in 143 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:12:58.922421 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-332d6b61-613e-4764-a6ee-5c3b46178201 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:58.936943 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-332d6b61-613e-4764-a6ee-5c3b46178201 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:58.943609 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9701c393-ee2b-46ce-afcf-a090cdd79b4e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:58.944692 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9701c393-ee2b-46ce-afcf-a090cdd79b4e admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:58.945182 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9701c393-ee2b-46ce-afcf-a090cdd79b4e admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:58.945721 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9701c393-ee2b-46ce-afcf-a090cdd79b4e admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:58.946535 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9701c393-ee2b-46ce-afcf-a090cdd79b4e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:58.949822 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9701c393-ee2b-46ce-afcf-a090cdd79b4e admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:58.966998 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1020/2013] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:12:58 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:12:58.972519 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb297354-f73a-494d-b8df-0d7b977f4da6 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:58.988698 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb297354-f73a-494d-b8df-0d7b977f4da6 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:59.006965 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-443fa83b-8f4e-4ae8-9db7-a3b490bc62c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:59.008369 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-443fa83b-8f4e-4ae8-9db7-a3b490bc62c0 admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:59.008859 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-443fa83b-8f4e-4ae8-9db7-a3b490bc62c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:59.009391 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-443fa83b-8f4e-4ae8-9db7-a3b490bc62c0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:59.011109 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-443fa83b-8f4e-4ae8-9db7-a3b490bc62c0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:59.014612 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-443fa83b-8f4e-4ae8-9db7-a3b490bc62c0 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:59.029208 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 994/2014] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:12:58 2022] GET /identity/v3/roles => generated 1420 bytes in 59 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:12:59.033097 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9701c393-ee2b-46ce-afcf-a090cdd79b4e admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:59.046958 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9701c393-ee2b-46ce-afcf-a090cdd79b4e admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:59.052180 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62fec960-07fb-45c6-b20c-cb72b3874866 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:59.052844 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62fec960-07fb-45c6-b20c-cb72b3874866 admin admin] REQUEST_METHOD: `PUT` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:59.053128 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62fec960-07fb-45c6-b20c-cb72b3874866 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:59.053401 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62fec960-07fb-45c6-b20c-cb72b3874866 admin admin] PATH_INFO: `/v3/projects/e1e846793de24f1496dadc7921868ce2/users/13fe4945eee44db1848a401d9f6b987b/roles/6f03d86980c34c36b44a8d6764ff4fda` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:59.070005 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62fec960-07fb-45c6-b20c-cb72b3874866 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e1e846793de24f1496dadc7921868ce2, user_id=13fe4945eee44db1848a401d9f6b987b, role_id=6f03d86980c34c36b44a8d6764ff4fda)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:59.072726 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62fec960-07fb-45c6-b20c-cb72b3874866 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:59.124118 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1021/2015] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:12:59 2022] PUT /identity/v3/projects/e1e846793de24f1496dadc7921868ce2/users/13fe4945eee44db1848a401d9f6b987b/roles/6f03d86980c34c36b44a8d6764ff4fda => generated 0 bytes in 80 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:12:59.124469 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfd8d608-e18b-45b6-ab57-c26650b19e38 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:59.124469 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfd8d608-e18b-45b6-ab57-c26650b19e38 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:59.124469 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfd8d608-e18b-45b6-ab57-c26650b19e38 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:59.158758 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-bfd8d608-e18b-45b6-ab57-c26650b19e38 None None] MFA Rules not processed for user `13fe4945eee44db1848a401d9f6b987b`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:12:59.160064 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-bfd8d608-e18b-45b6-ab57-c26650b19e38 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:59.180500 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:59.257155 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 995/2016] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:12:59 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 05:12:59.258707 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62fec960-07fb-45c6-b20c-cb72b3874866 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:59.264769 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=89,keystone:DELETE=8,keystone:INSERT=3 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:59.283327 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfd8d608-e18b-45b6-ab57-c26650b19e38 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:59.284986 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-bfd8d608-e18b-45b6-ab57-c26650b19e38 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:59.342012 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62fec960-07fb-45c6-b20c-cb72b3874866 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:59.352204 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51d7bf77-1a5a-47a8-9e44-6eb779dd4abf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:59.353232 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51d7bf77-1a5a-47a8-9e44-6eb779dd4abf admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:59.354368 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51d7bf77-1a5a-47a8-9e44-6eb779dd4abf admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:59.355184 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51d7bf77-1a5a-47a8-9e44-6eb779dd4abf admin admin] PATH_INFO: `/v3/projects` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:59.356035 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51d7bf77-1a5a-47a8-9e44-6eb779dd4abf admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:59.358360 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51d7bf77-1a5a-47a8-9e44-6eb779dd4abf admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:59.412875 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1022/2017] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:12:59 2022] POST /identity/v3/projects => generated 381 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:12:59.418794 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51d7bf77-1a5a-47a8-9e44-6eb779dd4abf admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:59.497822 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51d7bf77-1a5a-47a8-9e44-6eb779dd4abf admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:59.503712 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b71561e4-bac9-474c-b017-f5196c48cdd0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:59.504587 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b71561e4-bac9-474c-b017-f5196c48cdd0 admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:59.504975 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b71561e4-bac9-474c-b017-f5196c48cdd0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:59.506052 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b71561e4-bac9-474c-b017-f5196c48cdd0 admin admin] PATH_INFO: `/v3/users` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:59.506052 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b71561e4-bac9-474c-b017-f5196c48cdd0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:59.517200 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b71561e4-bac9-474c-b017-f5196c48cdd0 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:59.523972 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfd8d608-e18b-45b6-ab57-c26650b19e38 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:59.545503 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2968d369-7b36-4e2a-bd0e-818039a986b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:59.546358 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2968d369-7b36-4e2a-bd0e-818039a986b1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:59.546472 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2968d369-7b36-4e2a-bd0e-818039a986b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:59.546669 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2968d369-7b36-4e2a-bd0e-818039a986b1 admin admin] PATH_INFO: `/v3/projects/5d156e26b9ac4afa88354d61756f2068` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:59.550741 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2968d369-7b36-4e2a-bd0e-818039a986b1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5d156e26b9ac4afa88354d61756f2068)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:59.552117 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2968d369-7b36-4e2a-bd0e-818039a986b1 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:59.567736 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1023/2018] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:12:59 2022] POST /identity/v3/users => generated 297 bytes in 149 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:12:59.568900 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b71561e4-bac9-474c-b017-f5196c48cdd0 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:59.590968 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b71561e4-bac9-474c-b017-f5196c48cdd0 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:59.604825 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc50cfee-befc-4b94-90d7-fd0737d4f5c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:59.605456 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc50cfee-befc-4b94-90d7-fd0737d4f5c1 admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:59.605741 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc50cfee-befc-4b94-90d7-fd0737d4f5c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:59.606015 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc50cfee-befc-4b94-90d7-fd0737d4f5c1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:59.606667 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc50cfee-befc-4b94-90d7-fd0737d4f5c1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:59.612203 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc50cfee-befc-4b94-90d7-fd0737d4f5c1 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:59.641349 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1024/2019] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:12:59 2022] GET /identity/v3/roles => generated 1420 bytes in 73 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:12:59.645916 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc50cfee-befc-4b94-90d7-fd0737d4f5c1 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:12:59.666146 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-2968d369-7b36-4e2a-bd0e-818039a986b1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5d156e26b9ac4afa88354d61756f2068', 'request_id': 'req-2968d369-7b36-4e2a-bd0e-818039a986b1'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:59.671004 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc50cfee-befc-4b94-90d7-fd0737d4f5c1 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:12:59.681752 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-2968d369-7b36-4e2a-bd0e-818039a986b1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5d156e26b9ac4afa88354d61756f2068', 'request_id': 'req-2968d369-7b36-4e2a-bd0e-818039a986b1'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:59.710990 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-2968d369-7b36-4e2a-bd0e-818039a986b1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5d156e26b9ac4afa88354d61756f2068', 'request_id': 'req-2968d369-7b36-4e2a-bd0e-818039a986b1'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:59.749168 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-2968d369-7b36-4e2a-bd0e-818039a986b1 admin admin] The token cache is being invalidate because project 5d156e26b9ac4afa88354d61756f2068 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81624) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:12:59.749476 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-2968d369-7b36-4e2a-bd0e-818039a986b1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:59.751904 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-2968d369-7b36-4e2a-bd0e-818039a986b1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:12:59.759848 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 996/2020] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:12:59 2022] DELETE /identity/v3/projects/5d156e26b9ac4afa88354d61756f2068 => generated 0 bytes in 474 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:12:59.761735 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f720c793-2c7e-4ff6-8545-ee87737bcf8e None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:59.762021 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f720c793-2c7e-4ff6-8545-ee87737bcf8e None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:59.762311 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f720c793-2c7e-4ff6-8545-ee87737bcf8e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:59.791021 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-f720c793-2c7e-4ff6-8545-ee87737bcf8e None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:12:59.792288 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-f720c793-2c7e-4ff6-8545-ee87737bcf8e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:12:59.857138 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98a5d41b-3269-4a71-a951-36414768899f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:12:59.861242 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98a5d41b-3269-4a71-a951-36414768899f admin admin] REQUEST_METHOD: `PUT` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:59.861764 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98a5d41b-3269-4a71-a951-36414768899f admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:59.863561 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98a5d41b-3269-4a71-a951-36414768899f admin admin] PATH_INFO: `/v3/projects/8dedadaa140546a3ae15a8a5c330c187/users/180814bf17b14e5cbb5d39809a38621c/roles/9d9d5d2c270146488249b7ed943280db` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:12:59.905058 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:59.905924 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:12:59.916245 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98a5d41b-3269-4a71-a951-36414768899f admin admin] RBAC: Authorizing `identity:create_grant(project_id=8dedadaa140546a3ae15a8a5c330c187, user_id=180814bf17b14e5cbb5d39809a38621c, role_id=9d9d5d2c270146488249b7ed943280db)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:12:59.928424 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98a5d41b-3269-4a71-a951-36414768899f admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:12:59.957196 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 997/2021] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:12:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 196 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:12:59.969644 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02316c9f-9aa7-48e1-836f-12efc6a9ecea None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:12:59.970096 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02316c9f-9aa7-48e1-836f-12efc6a9ecea None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:12:59.970680 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02316c9f-9aa7-48e1-836f-12efc6a9ecea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:00.002918 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1025/2022] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:12:59 2022] PUT /identity/v3/projects/8dedadaa140546a3ae15a8a5c330c187/users/180814bf17b14e5cbb5d39809a38621c/roles/9d9d5d2c270146488249b7ed943280db => generated 0 bytes in 356 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:13:00.007258 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98a5d41b-3269-4a71-a951-36414768899f admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:00.010801 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-98a5d41b-3269-4a71-a951-36414768899f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:00.052307 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-02316c9f-9aa7-48e1-836f-12efc6a9ecea None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:13:00.053540 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-02316c9f-9aa7-48e1-836f-12efc6a9ecea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:00.148433 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 998/2023] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:12:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 05:13:00.153048 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98a5d41b-3269-4a71-a951-36414768899f admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:00.153988 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02316c9f-9aa7-48e1-836f-12efc6a9ecea None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:00.160074 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff60e560-72bc-40e9-982b-891b78eca373 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:00.160856 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff60e560-72bc-40e9-982b-891b78eca373 admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:00.161271 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff60e560-72bc-40e9-982b-891b78eca373 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:00.161548 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff60e560-72bc-40e9-982b-891b78eca373 admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:00.162205 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff60e560-72bc-40e9-982b-891b78eca373 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:00.164023 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff60e560-72bc-40e9-982b-891b78eca373 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:00.177459 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02316c9f-9aa7-48e1-836f-12efc6a9ecea None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:00.184644 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60071a77-769b-48b5-9cdf-1f69e0f283a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:00.185757 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60071a77-769b-48b5-9cdf-1f69e0f283a4 admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:00.186038 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60071a77-769b-48b5-9cdf-1f69e0f283a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:00.186306 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60071a77-769b-48b5-9cdf-1f69e0f283a4 admin admin] PATH_INFO: `/v3/domains` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:00.187075 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60071a77-769b-48b5-9cdf-1f69e0f283a4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:13:00.187386 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60071a77-769b-48b5-9cdf-1f69e0f283a4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:00.189352 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60071a77-769b-48b5-9cdf-1f69e0f283a4 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:00.198396 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1026/2024] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:13:00 2022] GET /identity/v3/roles => generated 1420 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:13:00.199525 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff60e560-72bc-40e9-982b-891b78eca373 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:00.208021 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 999/2025] 10.222.0.77 () {58 vars in 1153 bytes} [Tue Aug 30 05:13:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:13:00.213243 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60071a77-769b-48b5-9cdf-1f69e0f283a4 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:00.215792 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff60e560-72bc-40e9-982b-891b78eca373 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:00.222079 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b7c41b4-811c-40e1-8850-2d2016d03e88 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:00.223001 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b7c41b4-811c-40e1-8850-2d2016d03e88 admin admin] REQUEST_METHOD: `PUT` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:00.223430 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b7c41b4-811c-40e1-8850-2d2016d03e88 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:00.223966 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b7c41b4-811c-40e1-8850-2d2016d03e88 admin admin] PATH_INFO: `/v3/projects/8dedadaa140546a3ae15a8a5c330c187/users/180814bf17b14e5cbb5d39809a38621c/roles/9d9d5d2c270146488249b7ed943280db` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:00.226112 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b7c41b4-811c-40e1-8850-2d2016d03e88 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8dedadaa140546a3ae15a8a5c330c187, user_id=180814bf17b14e5cbb5d39809a38621c, role_id=9d9d5d2c270146488249b7ed943280db)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:00.229830 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b7c41b4-811c-40e1-8850-2d2016d03e88 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:00.230803 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60071a77-769b-48b5-9cdf-1f69e0f283a4 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:00.238653 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-753845ad-b8d5-46f1-aa0d-df841629a52f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:00.239587 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-753845ad-b8d5-46f1-aa0d-df841629a52f admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:00.239991 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-753845ad-b8d5-46f1-aa0d-df841629a52f admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:00.240361 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-753845ad-b8d5-46f1-aa0d-df841629a52f admin admin] PATH_INFO: `/v3/projects` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:00.241092 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-753845ad-b8d5-46f1-aa0d-df841629a52f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:00.244101 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-753845ad-b8d5-46f1-aa0d-df841629a52f admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:00.246195 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1027/2026] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:13:00 2022] PUT /identity/v3/projects/8dedadaa140546a3ae15a8a5c330c187/users/180814bf17b14e5cbb5d39809a38621c/roles/9d9d5d2c270146488249b7ed943280db => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:13:00.257311 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f488881-e2b6-471a-a549-6930e22b4d90 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:00.257706 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f488881-e2b6-471a-a549-6930e22b4d90 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:00.258153 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f488881-e2b6-471a-a549-6930e22b4d90 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:00.298469 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1000/2027] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:13:00 2022] POST /identity/v3/projects => generated 381 bytes in 86 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:13:00.300370 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-753845ad-b8d5-46f1-aa0d-df841629a52f admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:00.303114 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-0f488881-e2b6-471a-a549-6930e22b4d90 None None] MFA Rules not processed for user `180814bf17b14e5cbb5d39809a38621c`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:13:00.305426 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-0f488881-e2b6-471a-a549-6930e22b4d90 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:00.419604 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-753845ad-b8d5-46f1-aa0d-df841629a52f admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:00.421591 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1028/2028] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:13:00 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:13:00.428260 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e62daa5-13d4-46ec-9a42-42d511cf9ed5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:00.428874 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e62daa5-13d4-46ec-9a42-42d511cf9ed5 admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:00.429150 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e62daa5-13d4-46ec-9a42-42d511cf9ed5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:00.429465 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e62daa5-13d4-46ec-9a42-42d511cf9ed5 admin admin] PATH_INFO: `/v3/users` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:00.430283 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e62daa5-13d4-46ec-9a42-42d511cf9ed5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:00.431468 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f488881-e2b6-471a-a549-6930e22b4d90 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:00.432537 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e62daa5-13d4-46ec-9a42-42d511cf9ed5 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:00.433843 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-0f488881-e2b6-471a-a549-6930e22b4d90 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:00.473165 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1001/2029] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:13:00 2022] POST /identity/v3/users => generated 291 bytes in 174 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:13:00.477587 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e62daa5-13d4-46ec-9a42-42d511cf9ed5 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:00.511829 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e62daa5-13d4-46ec-9a42-42d511cf9ed5 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:00.523273 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f488881-e2b6-471a-a549-6930e22b4d90 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:00.527819 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae115483-f1cc-4222-ba7b-506a8e946201 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:00.528729 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae115483-f1cc-4222-ba7b-506a8e946201 admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:00.529079 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae115483-f1cc-4222-ba7b-506a8e946201 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:00.529407 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae115483-f1cc-4222-ba7b-506a8e946201 admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:00.530353 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae115483-f1cc-4222-ba7b-506a8e946201 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:00.535122 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae115483-f1cc-4222-ba7b-506a8e946201 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:00.536524 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de332612-b7f3-45a9-8c71-fb53a22826e7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '019a79b2affe478b8a30d1c538d0415c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:00.537574 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de332612-b7f3-45a9-8c71-fb53a22826e7 service cinder] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:00.538002 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de332612-b7f3-45a9-8c71-fb53a22826e7 service cinder] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:00.538568 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de332612-b7f3-45a9-8c71-fb53a22826e7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:00.558765 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de332612-b7f3-45a9-8c71-fb53a22826e7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:00.567073 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de332612-b7f3-45a9-8c71-fb53a22826e7 service cinder] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:00.570902 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1002/2030] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:13:00 2022] GET /identity/v3/roles => generated 1420 bytes in 94 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 05:13:00.571949 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae115483-f1cc-4222-ba7b-506a8e946201 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:00.590644 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1029/2031] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:13:00 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 160 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:13:00.593177 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae115483-f1cc-4222-ba7b-506a8e946201 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:00.621959 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65dd4afb-d4b8-45e6-a8ec-eab017185ce3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:00.622792 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65dd4afb-d4b8-45e6-a8ec-eab017185ce3 admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:00.623213 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65dd4afb-d4b8-45e6-a8ec-eab017185ce3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:00.623545 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65dd4afb-d4b8-45e6-a8ec-eab017185ce3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:00.624770 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65dd4afb-d4b8-45e6-a8ec-eab017185ce3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:00.636048 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65dd4afb-d4b8-45e6-a8ec-eab017185ce3 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:00.658074 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de332612-b7f3-45a9-8c71-fb53a22826e7 service cinder] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:00.660054 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1003/2032] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:13:00 2022] GET /identity/v3/roles => generated 1420 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:13:00.664681 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:00.674092 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de332612-b7f3-45a9-8c71-fb53a22826e7 service cinder] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:00.682710 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7505976-b953-438b-b3e6-f932ca7a062d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:00.683553 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7505976-b953-438b-b3e6-f932ca7a062d admin admin] REQUEST_METHOD: `PUT` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:00.684390 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7505976-b953-438b-b3e6-f932ca7a062d admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:00.684808 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7505976-b953-438b-b3e6-f932ca7a062d admin admin] PATH_INFO: `/v3/projects/2a2b66d94ffc4e87804d537f1f0921b2/users/9798192e9308461891b140de184ee354/roles/6f03d86980c34c36b44a8d6764ff4fda` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:00.725966 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7505976-b953-438b-b3e6-f932ca7a062d admin admin] RBAC: Authorizing `identity:create_grant(project_id=2a2b66d94ffc4e87804d537f1f0921b2, user_id=9798192e9308461891b140de184ee354, role_id=6f03d86980c34c36b44a8d6764ff4fda)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:00.731816 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7505976-b953-438b-b3e6-f932ca7a062d admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:00.787509 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1030/2033] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:13:00 2022] PUT /identity/v3/projects/2a2b66d94ffc4e87804d537f1f0921b2/users/9798192e9308461891b140de184ee354/roles/6f03d86980c34c36b44a8d6764ff4fda => generated 0 bytes in 130 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:13:00.800747 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dffc941a-5791-4363-a8a6-62a7bc305017 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:00.800747 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dffc941a-5791-4363-a8a6-62a7bc305017 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:00.800747 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dffc941a-5791-4363-a8a6-62a7bc305017 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:00.857834 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-dffc941a-5791-4363-a8a6-62a7bc305017 None None] MFA Rules not processed for user `9798192e9308461891b140de184ee354`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:13:00.859579 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-dffc941a-5791-4363-a8a6-62a7bc305017 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:00.887892 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:00.991493 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1004/2034] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:13:00 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:13:00.996352 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7505976-b953-438b-b3e6-f932ca7a062d admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:01.139889 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7505976-b953-438b-b3e6-f932ca7a062d admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:01.154996 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c96b2ae-537f-4e63-aeb9-cc811bf6b2cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:01.156166 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c96b2ae-537f-4e63-aeb9-cc811bf6b2cc admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:01.157106 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c96b2ae-537f-4e63-aeb9-cc811bf6b2cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:01.158208 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c96b2ae-537f-4e63-aeb9-cc811bf6b2cc admin admin] PATH_INFO: `/v3/projects` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:01.159653 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c96b2ae-537f-4e63-aeb9-cc811bf6b2cc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:01.163098 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c96b2ae-537f-4e63-aeb9-cc811bf6b2cc admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:01.164933 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dffc941a-5791-4363-a8a6-62a7bc305017 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:01.292246 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1031/2035] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:13:00 2022] POST /identity/v3/projects => generated 383 bytes in 294 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:13:01.295744 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c96b2ae-537f-4e63-aeb9-cc811bf6b2cc admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:01.354038 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dffc941a-5791-4363-a8a6-62a7bc305017 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:01.470925 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c96b2ae-537f-4e63-aeb9-cc811bf6b2cc admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:01.479540 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8077fbc5-4067-4484-a1d9-9a95acc1d065 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:01.479540 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8077fbc5-4067-4484-a1d9-9a95acc1d065 admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:01.479923 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8077fbc5-4067-4484-a1d9-9a95acc1d065 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:01.480290 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8077fbc5-4067-4484-a1d9-9a95acc1d065 admin admin] PATH_INFO: `/v3/users` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:01.481019 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8077fbc5-4067-4484-a1d9-9a95acc1d065 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:01.483516 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8077fbc5-4067-4484-a1d9-9a95acc1d065 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:01.517723 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deb65b6c-31a6-4d23-a10b-f0ab7a10e249 tempest-VolumeQuotaClassesTest-1603300585 tempest-VolumeQuotaClassesTest-1603300585-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '180814bf17b14e5cbb5d39809a38621c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8dedadaa140546a3ae15a8a5c330c187', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:01.519115 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deb65b6c-31a6-4d23-a10b-f0ab7a10e249 tempest-VolumeQuotaClassesTest-1603300585 tempest-VolumeQuotaClassesTest-1603300585-project-admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:01.519398 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deb65b6c-31a6-4d23-a10b-f0ab7a10e249 tempest-VolumeQuotaClassesTest-1603300585 tempest-VolumeQuotaClassesTest-1603300585-project-admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:01.519722 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deb65b6c-31a6-4d23-a10b-f0ab7a10e249 tempest-VolumeQuotaClassesTest-1603300585 tempest-VolumeQuotaClassesTest-1603300585-project-admin] PATH_INFO: `/v3/domains` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:01.520382 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deb65b6c-31a6-4d23-a10b-f0ab7a10e249 tempest-VolumeQuotaClassesTest-1603300585 tempest-VolumeQuotaClassesTest-1603300585-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:13:01.520690 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deb65b6c-31a6-4d23-a10b-f0ab7a10e249 tempest-VolumeQuotaClassesTest-1603300585 tempest-VolumeQuotaClassesTest-1603300585-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:01.522562 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deb65b6c-31a6-4d23-a10b-f0ab7a10e249 tempest-VolumeQuotaClassesTest-1603300585 tempest-VolumeQuotaClassesTest-1603300585-project-admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:01.537739 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1032/2036] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:13:01 2022] POST /identity/v3/users => generated 298 bytes in 242 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:13:01.539252 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8077fbc5-4067-4484-a1d9-9a95acc1d065 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:01.549172 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1005/2037] 10.222.0.77 () {58 vars in 1153 bytes} [Tue Aug 30 05:13:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 384 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:13:01.550078 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deb65b6c-31a6-4d23-a10b-f0ab7a10e249 tempest-VolumeQuotaClassesTest-1603300585 tempest-VolumeQuotaClassesTest-1603300585-project-admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:01.551917 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8077fbc5-4067-4484-a1d9-9a95acc1d065 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:01.557171 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3690759-f8f4-47ca-aa61-c983b3cbc8f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:01.558297 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3690759-f8f4-47ca-aa61-c983b3cbc8f1 admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:01.558857 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3690759-f8f4-47ca-aa61-c983b3cbc8f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:01.559362 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3690759-f8f4-47ca-aa61-c983b3cbc8f1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:01.560161 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3690759-f8f4-47ca-aa61-c983b3cbc8f1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:01.561785 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deb65b6c-31a6-4d23-a10b-f0ab7a10e249 tempest-VolumeQuotaClassesTest-1603300585 tempest-VolumeQuotaClassesTest-1603300585-project-admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:01.562006 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3690759-f8f4-47ca-aa61-c983b3cbc8f1 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:01.568029 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e241dbaf-10fa-4d05-8ef9-907804aa42a3 tempest-VolumeQuotaClassesTest-1603300585 tempest-VolumeQuotaClassesTest-1603300585-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '180814bf17b14e5cbb5d39809a38621c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8dedadaa140546a3ae15a8a5c330c187', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:01.568866 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e241dbaf-10fa-4d05-8ef9-907804aa42a3 tempest-VolumeQuotaClassesTest-1603300585 tempest-VolumeQuotaClassesTest-1603300585-project-admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:01.569318 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e241dbaf-10fa-4d05-8ef9-907804aa42a3 tempest-VolumeQuotaClassesTest-1603300585 tempest-VolumeQuotaClassesTest-1603300585-project-admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:01.569596 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e241dbaf-10fa-4d05-8ef9-907804aa42a3 tempest-VolumeQuotaClassesTest-1603300585 tempest-VolumeQuotaClassesTest-1603300585-project-admin] PATH_INFO: `/v3/projects` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:01.570286 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e241dbaf-10fa-4d05-8ef9-907804aa42a3 tempest-VolumeQuotaClassesTest-1603300585 tempest-VolumeQuotaClassesTest-1603300585-project-admin] RBAC: Authorizing `identity:create_project()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:01.572236 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e241dbaf-10fa-4d05-8ef9-907804aa42a3 tempest-VolumeQuotaClassesTest-1603300585 tempest-VolumeQuotaClassesTest-1603300585-project-admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:01.575849 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1033/2038] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:13:01 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:13:01.581048 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3690759-f8f4-47ca-aa61-c983b3cbc8f1 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:01.597396 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3690759-f8f4-47ca-aa61-c983b3cbc8f1 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:01.606742 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-381f36d7-67ca-4ed7-9dc9-70e661e7216c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:01.608664 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-381f36d7-67ca-4ed7-9dc9-70e661e7216c admin admin] REQUEST_METHOD: `PUT` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:01.609530 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-381f36d7-67ca-4ed7-9dc9-70e661e7216c admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:01.609920 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-381f36d7-67ca-4ed7-9dc9-70e661e7216c admin admin] PATH_INFO: `/v3/projects/2bccc550dc9d4980ad456b76724602ef/users/57978e4009984afaaf362947600f8497/roles/9d9d5d2c270146488249b7ed943280db` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:01.645521 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-381f36d7-67ca-4ed7-9dc9-70e661e7216c admin admin] RBAC: Authorizing `identity:create_grant(project_id=2bccc550dc9d4980ad456b76724602ef, user_id=57978e4009984afaaf362947600f8497, role_id=9d9d5d2c270146488249b7ed943280db)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:01.649623 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-381f36d7-67ca-4ed7-9dc9-70e661e7216c admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:01.653133 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1006/2039] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:13:01 2022] POST /identity/v3/projects => generated 354 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:13:01.654186 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e241dbaf-10fa-4d05-8ef9-907804aa42a3 tempest-VolumeQuotaClassesTest-1603300585 tempest-VolumeQuotaClassesTest-1603300585-project-admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:01.676128 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1034/2040] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:13:01 2022] PUT /identity/v3/projects/2bccc550dc9d4980ad456b76724602ef/users/57978e4009984afaaf362947600f8497/roles/9d9d5d2c270146488249b7ed943280db => generated 0 bytes in 96 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:13:01.683266 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-381f36d7-67ca-4ed7-9dc9-70e661e7216c admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:01.714446 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:01.750910 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:01.798393 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-381f36d7-67ca-4ed7-9dc9-70e661e7216c admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:01.805591 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e241dbaf-10fa-4d05-8ef9-907804aa42a3 tempest-VolumeQuotaClassesTest-1603300585 tempest-VolumeQuotaClassesTest-1603300585-project-admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:01.811437 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c7e58a7-2609-4791-9dce-1b3709a04865 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:01.814011 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c7e58a7-2609-4791-9dce-1b3709a04865 admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:01.814482 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c7e58a7-2609-4791-9dce-1b3709a04865 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:01.815501 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c7e58a7-2609-4791-9dce-1b3709a04865 admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:01.816323 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c7e58a7-2609-4791-9dce-1b3709a04865 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:01.818472 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c7e58a7-2609-4791-9dce-1b3709a04865 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:01.833391 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1035/2041] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:13:01 2022] GET /identity/v3/roles => generated 1420 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:13:01.837892 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c7e58a7-2609-4791-9dce-1b3709a04865 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:01.853729 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c7e58a7-2609-4791-9dce-1b3709a04865 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:01.869069 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c297f29a-22c5-4f7c-9820-0d7ab23e026d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:01.870574 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c297f29a-22c5-4f7c-9820-0d7ab23e026d admin admin] REQUEST_METHOD: `PUT` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:01.871010 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c297f29a-22c5-4f7c-9820-0d7ab23e026d admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:01.871348 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c297f29a-22c5-4f7c-9820-0d7ab23e026d admin admin] PATH_INFO: `/v3/projects/2bccc550dc9d4980ad456b76724602ef/users/57978e4009984afaaf362947600f8497/roles/9d9d5d2c270146488249b7ed943280db` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:01.873839 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c297f29a-22c5-4f7c-9820-0d7ab23e026d admin admin] RBAC: Authorizing `identity:create_grant(project_id=2bccc550dc9d4980ad456b76724602ef, user_id=57978e4009984afaaf362947600f8497, role_id=9d9d5d2c270146488249b7ed943280db)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:01.876784 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c297f29a-22c5-4f7c-9820-0d7ab23e026d admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:01.890656 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1036/2042] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:13:01 2022] PUT /identity/v3/projects/2bccc550dc9d4980ad456b76724602ef/users/57978e4009984afaaf362947600f8497/roles/9d9d5d2c270146488249b7ed943280db => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:13:01.901033 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-698bb946-cb69-41ba-9dd6-17373043c46c None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:01.901426 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-698bb946-cb69-41ba-9dd6-17373043c46c None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:01.901735 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-698bb946-cb69-41ba-9dd6-17373043c46c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:01.911510 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5065e93-62ef-4d8b-8b25-883080c71ba2 tempest-VolumeQuotaClassesTest-1603300585 tempest-VolumeQuotaClassesTest-1603300585-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '180814bf17b14e5cbb5d39809a38621c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8dedadaa140546a3ae15a8a5c330c187', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:01.912125 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5065e93-62ef-4d8b-8b25-883080c71ba2 tempest-VolumeQuotaClassesTest-1603300585 tempest-VolumeQuotaClassesTest-1603300585-project-admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:01.912406 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5065e93-62ef-4d8b-8b25-883080c71ba2 tempest-VolumeQuotaClassesTest-1603300585 tempest-VolumeQuotaClassesTest-1603300585-project-admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:01.912680 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5065e93-62ef-4d8b-8b25-883080c71ba2 tempest-VolumeQuotaClassesTest-1603300585 tempest-VolumeQuotaClassesTest-1603300585-project-admin] PATH_INFO: `/v3/domains` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:01.913667 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5065e93-62ef-4d8b-8b25-883080c71ba2 tempest-VolumeQuotaClassesTest-1603300585 tempest-VolumeQuotaClassesTest-1603300585-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:13:01.913774 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5065e93-62ef-4d8b-8b25-883080c71ba2 tempest-VolumeQuotaClassesTest-1603300585 tempest-VolumeQuotaClassesTest-1603300585-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:01.915777 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5065e93-62ef-4d8b-8b25-883080c71ba2 tempest-VolumeQuotaClassesTest-1603300585 tempest-VolumeQuotaClassesTest-1603300585-project-admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:01.928457 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1007/2043] 10.222.0.77 () {58 vars in 1153 bytes} [Tue Aug 30 05:13:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 275 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:13:01.933995 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-698bb946-cb69-41ba-9dd6-17373043c46c None None] MFA Rules not processed for user `57978e4009984afaaf362947600f8497`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:13:01.935312 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-698bb946-cb69-41ba-9dd6-17373043c46c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:01.970435 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5065e93-62ef-4d8b-8b25-883080c71ba2 tempest-VolumeQuotaClassesTest-1603300585 tempest-VolumeQuotaClassesTest-1603300585-project-admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:01.982747 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5065e93-62ef-4d8b-8b25-883080c71ba2 tempest-VolumeQuotaClassesTest-1603300585 tempest-VolumeQuotaClassesTest-1603300585-project-admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:02.027672 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1037/2044] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:13:01 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 05:13:02.042748 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-698bb946-cb69-41ba-9dd6-17373043c46c None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:02.089165 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-172fb56b-abb8-4b74-8103-82aae48d32ce tempest-VolumeQuotaClassesTest-1603300585 tempest-VolumeQuotaClassesTest-1603300585-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '180814bf17b14e5cbb5d39809a38621c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8dedadaa140546a3ae15a8a5c330c187', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:02.090038 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-172fb56b-abb8-4b74-8103-82aae48d32ce tempest-VolumeQuotaClassesTest-1603300585 tempest-VolumeQuotaClassesTest-1603300585-project-admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:02.091080 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-172fb56b-abb8-4b74-8103-82aae48d32ce tempest-VolumeQuotaClassesTest-1603300585 tempest-VolumeQuotaClassesTest-1603300585-project-admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:02.091242 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-172fb56b-abb8-4b74-8103-82aae48d32ce tempest-VolumeQuotaClassesTest-1603300585 tempest-VolumeQuotaClassesTest-1603300585-project-admin] PATH_INFO: `/v3/projects/af8b1317c4bf4aa5991e1b3d32ac30fa` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:02.096502 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-172fb56b-abb8-4b74-8103-82aae48d32ce tempest-VolumeQuotaClassesTest-1603300585 tempest-VolumeQuotaClassesTest-1603300585-project-admin] RBAC: Authorizing `identity:delete_project(project_id=af8b1317c4bf4aa5991e1b3d32ac30fa)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:02.099890 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-172fb56b-abb8-4b74-8103-82aae48d32ce tempest-VolumeQuotaClassesTest-1603300585 tempest-VolumeQuotaClassesTest-1603300585-project-admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:02.138278 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-698bb946-cb69-41ba-9dd6-17373043c46c None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:02.144147 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19797abe-6063-4268-a633-abb990b15ef6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '019a79b2affe478b8a30d1c538d0415c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:02.144987 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19797abe-6063-4268-a633-abb990b15ef6 service cinder] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:02.145399 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19797abe-6063-4268-a633-abb990b15ef6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:02.145826 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19797abe-6063-4268-a633-abb990b15ef6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:02.162519 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19797abe-6063-4268-a633-abb990b15ef6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:02.164979 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19797abe-6063-4268-a633-abb990b15ef6 service cinder] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:02.177259 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-172fb56b-abb8-4b74-8103-82aae48d32ce tempest-VolumeQuotaClassesTest-1603300585 tempest-VolumeQuotaClassesTest-1603300585-project-admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'af8b1317c4bf4aa5991e1b3d32ac30fa', 'request_id': 'req-172fb56b-abb8-4b74-8103-82aae48d32ce'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:02.182384 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1038/2045] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:13:02 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 140 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:13:02.186333 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-172fb56b-abb8-4b74-8103-82aae48d32ce tempest-VolumeQuotaClassesTest-1603300585 tempest-VolumeQuotaClassesTest-1603300585-project-admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'af8b1317c4bf4aa5991e1b3d32ac30fa', 'request_id': 'req-172fb56b-abb8-4b74-8103-82aae48d32ce'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:02.195601 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-172fb56b-abb8-4b74-8103-82aae48d32ce tempest-VolumeQuotaClassesTest-1603300585 tempest-VolumeQuotaClassesTest-1603300585-project-admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'af8b1317c4bf4aa5991e1b3d32ac30fa', 'request_id': 'req-172fb56b-abb8-4b74-8103-82aae48d32ce'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:02.208733 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-172fb56b-abb8-4b74-8103-82aae48d32ce tempest-VolumeQuotaClassesTest-1603300585 tempest-VolumeQuotaClassesTest-1603300585-project-admin] The token cache is being invalidate because project af8b1317c4bf4aa5991e1b3d32ac30fa was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81624) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:13:02.209011 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-172fb56b-abb8-4b74-8103-82aae48d32ce tempest-VolumeQuotaClassesTest-1603300585 tempest-VolumeQuotaClassesTest-1603300585-project-admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:02.209734 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-172fb56b-abb8-4b74-8103-82aae48d32ce tempest-VolumeQuotaClassesTest-1603300585 tempest-VolumeQuotaClassesTest-1603300585-project-admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:02.211357 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1008/2046] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:13:01 2022] DELETE /identity/v3/projects/af8b1317c4bf4aa5991e1b3d32ac30fa => generated 0 bytes in 242 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:13:02.339914 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19797abe-6063-4268-a633-abb990b15ef6 service cinder] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:02.350535 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19797abe-6063-4268-a633-abb990b15ef6 service cinder] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:02.386387 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:02.415486 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfe8ee56-3417-4f14-a276-a7645661bfac service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '019a79b2affe478b8a30d1c538d0415c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:02.416083 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfe8ee56-3417-4f14-a276-a7645661bfac service cinder] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:02.416358 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfe8ee56-3417-4f14-a276-a7645661bfac service cinder] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:02.416630 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfe8ee56-3417-4f14-a276-a7645661bfac service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:02.418360 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-dfe8ee56-3417-4f14-a276-a7645661bfac service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:02.496795 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-172fb56b-abb8-4b74-8103-82aae48d32ce tempest-VolumeQuotaClassesTest-1603300585 tempest-VolumeQuotaClassesTest-1603300585-project-admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:02.498535 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfe8ee56-3417-4f14-a276-a7645661bfac service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:02.499009 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-172fb56b-abb8-4b74-8103-82aae48d32ce tempest-VolumeQuotaClassesTest-1603300585 tempest-VolumeQuotaClassesTest-1603300585-project-admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:02.500686 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfe8ee56-3417-4f14-a276-a7645661bfac service cinder] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:02.508473 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:02.515290 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1039/2047] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:13:02 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 176 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 05:13:02.620819 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-172fb56b-abb8-4b74-8103-82aae48d32ce tempest-VolumeQuotaClassesTest-1603300585 tempest-VolumeQuotaClassesTest-1603300585-project-admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:02.627078 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30c7c39f-a27b-446c-97db-1168e493df9a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:02.628363 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30c7c39f-a27b-446c-97db-1168e493df9a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:02.628651 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30c7c39f-a27b-446c-97db-1168e493df9a admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:02.628928 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30c7c39f-a27b-446c-97db-1168e493df9a admin admin] PATH_INFO: `/v3/users/13fe4945eee44db1848a401d9f6b987b` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:02.630157 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30c7c39f-a27b-446c-97db-1168e493df9a admin admin] RBAC: Authorizing `identity:delete_user(user_id=13fe4945eee44db1848a401d9f6b987b)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:02.632125 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30c7c39f-a27b-446c-97db-1168e493df9a admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:02.704431 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-30c7c39f-a27b-446c-97db-1168e493df9a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '13fe4945eee44db1848a401d9f6b987b'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:02.705396 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-30c7c39f-a27b-446c-97db-1168e493df9a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '13fe4945eee44db1848a401d9f6b987b'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:02.706200 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-30c7c39f-a27b-446c-97db-1168e493df9a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '13fe4945eee44db1848a401d9f6b987b'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:02.722971 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-30c7c39f-a27b-446c-97db-1168e493df9a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '13fe4945eee44db1848a401d9f6b987b'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:02.730783 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-30c7c39f-a27b-446c-97db-1168e493df9a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '13fe4945eee44db1848a401d9f6b987b'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:02.750781 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1009/2048] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:13:02 2022] DELETE /identity/v3/users/13fe4945eee44db1848a401d9f6b987b => generated 0 bytes in 254 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 05:13:02.755287 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3c4a26a-73e1-4a8b-b44f-55e5a6ef5ecc None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:02.756248 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3c4a26a-73e1-4a8b-b44f-55e5a6ef5ecc None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:02.757727 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3c4a26a-73e1-4a8b-b44f-55e5a6ef5ecc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:02.785905 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-a3c4a26a-73e1-4a8b-b44f-55e5a6ef5ecc None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:13:02.787190 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-a3c4a26a-73e1-4a8b-b44f-55e5a6ef5ecc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:02.880552 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30c7c39f-a27b-446c-97db-1168e493df9a admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:02.883806 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-30c7c39f-a27b-446c-97db-1168e493df9a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:02.891292 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1040/2049] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:13:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:13:02.897810 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3c4a26a-73e1-4a8b-b44f-55e5a6ef5ecc None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:02.899230 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-a3c4a26a-73e1-4a8b-b44f-55e5a6ef5ecc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:02.906125 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30c7c39f-a27b-446c-97db-1168e493df9a admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:02.911526 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58cf5233-51a3-47d3-ad48-c587a6fe51a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:02.912450 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58cf5233-51a3-47d3-ad48-c587a6fe51a3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:02.912806 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58cf5233-51a3-47d3-ad48-c587a6fe51a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:02.913129 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58cf5233-51a3-47d3-ad48-c587a6fe51a3 admin admin] PATH_INFO: `/v3/users/9798192e9308461891b140de184ee354` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:02.914425 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58cf5233-51a3-47d3-ad48-c587a6fe51a3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9798192e9308461891b140de184ee354)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:02.917087 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58cf5233-51a3-47d3-ad48-c587a6fe51a3 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:02.970202 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-58cf5233-51a3-47d3-ad48-c587a6fe51a3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9798192e9308461891b140de184ee354'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:02.971570 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-58cf5233-51a3-47d3-ad48-c587a6fe51a3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9798192e9308461891b140de184ee354'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:02.972046 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3c4a26a-73e1-4a8b-b44f-55e5a6ef5ecc None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:02.972774 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-58cf5233-51a3-47d3-ad48-c587a6fe51a3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9798192e9308461891b140de184ee354'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:02.976577 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46281b23-50df-486e-abcf-9e5b14daa55f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:02.977251 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46281b23-50df-486e-abcf-9e5b14daa55f service neutron] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:02.977531 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46281b23-50df-486e-abcf-9e5b14daa55f service neutron] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:02.977827 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46281b23-50df-486e-abcf-9e5b14daa55f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:02.980024 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-46281b23-50df-486e-abcf-9e5b14daa55f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:02.988262 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-58cf5233-51a3-47d3-ad48-c587a6fe51a3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9798192e9308461891b140de184ee354'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:02.992590 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46281b23-50df-486e-abcf-9e5b14daa55f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:02.994008 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46281b23-50df-486e-abcf-9e5b14daa55f service neutron] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:02.995084 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-58cf5233-51a3-47d3-ad48-c587a6fe51a3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9798192e9308461891b140de184ee354'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:02.999088 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=78,keystone:DELETE=20,keystone:INSERT=2 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:03.004880 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1041/2050] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:13:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 05:13:03.005693 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46281b23-50df-486e-abcf-9e5b14daa55f service neutron] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:03.007140 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-46281b23-50df-486e-abcf-9e5b14daa55f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:03.018174 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1010/2051] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:13:02 2022] DELETE /identity/v3/users/9798192e9308461891b140de184ee354 => generated 0 bytes in 138 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:13:03.020887 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46281b23-50df-486e-abcf-9e5b14daa55f service neutron] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:03.023429 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf7162c0-43a7-4549-8726-b2231e322f44 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:03.023720 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf7162c0-43a7-4549-8726-b2231e322f44 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:03.023994 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf7162c0-43a7-4549-8726-b2231e322f44 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:03.047722 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-cf7162c0-43a7-4549-8726-b2231e322f44 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:13:03.048934 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:03.049688 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-cf7162c0-43a7-4549-8726-b2231e322f44 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:03.120105 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8838646a-45bb-4951-99ee-e31a01dd601b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:03.120906 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8838646a-45bb-4951-99ee-e31a01dd601b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:03.121739 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8838646a-45bb-4951-99ee-e31a01dd601b admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:03.122098 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8838646a-45bb-4951-99ee-e31a01dd601b admin admin] PATH_INFO: `/v3/users/8589cbfccb2d4b0b9e294bcf4583edb3` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:03.123358 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8838646a-45bb-4951-99ee-e31a01dd601b admin admin] RBAC: Authorizing `identity:delete_user(user_id=8589cbfccb2d4b0b9e294bcf4583edb3)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:03.125733 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8838646a-45bb-4951-99ee-e31a01dd601b admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:03.144218 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1011/2052] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:13:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:13:03.150753 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf7162c0-43a7-4549-8726-b2231e322f44 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:03.153021 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-cf7162c0-43a7-4549-8726-b2231e322f44 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:03.198444 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-8838646a-45bb-4951-99ee-e31a01dd601b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8589cbfccb2d4b0b9e294bcf4583edb3'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:03.208472 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-8838646a-45bb-4951-99ee-e31a01dd601b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8589cbfccb2d4b0b9e294bcf4583edb3'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:03.224962 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf7162c0-43a7-4549-8726-b2231e322f44 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:03.229316 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-478dc08a-ce41-4da2-b5e5-26cde01f7149 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:03.229912 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478dc08a-ce41-4da2-b5e5-26cde01f7149 service neutron] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:03.230195 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478dc08a-ce41-4da2-b5e5-26cde01f7149 service neutron] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:03.230465 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478dc08a-ce41-4da2-b5e5-26cde01f7149 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:03.232172 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-8838646a-45bb-4951-99ee-e31a01dd601b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8589cbfccb2d4b0b9e294bcf4583edb3'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:03.240103 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-8838646a-45bb-4951-99ee-e31a01dd601b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8589cbfccb2d4b0b9e294bcf4583edb3'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:03.240419 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-8838646a-45bb-4951-99ee-e31a01dd601b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8589cbfccb2d4b0b9e294bcf4583edb3'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:03.241163 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-478dc08a-ce41-4da2-b5e5-26cde01f7149 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:03.242665 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-478dc08a-ce41-4da2-b5e5-26cde01f7149 service neutron] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:03.249272 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1042/2053] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:13:03 2022] DELETE /identity/v3/users/8589cbfccb2d4b0b9e294bcf4583edb3 => generated 0 bytes in 241 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:13:03.250765 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-630111bf-2347-4e6c-ab4f-5b3ecc339c6e None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:03.251143 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-630111bf-2347-4e6c-ab4f-5b3ecc339c6e None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:03.251894 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-630111bf-2347-4e6c-ab4f-5b3ecc339c6e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:03.257062 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1012/2054] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:13:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 05:13:03.273028 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-630111bf-2347-4e6c-ab4f-5b3ecc339c6e None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:13:03.274311 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-630111bf-2347-4e6c-ab4f-5b3ecc339c6e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:03.345189 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1043/2055] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:13:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 05:13:03.348270 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-478dc08a-ce41-4da2-b5e5-26cde01f7149 service neutron] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:03.349921 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-478dc08a-ce41-4da2-b5e5-26cde01f7149 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:03.356523 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-630111bf-2347-4e6c-ab4f-5b3ecc339c6e None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:03.358020 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-630111bf-2347-4e6c-ab4f-5b3ecc339c6e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:03.364133 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-478dc08a-ce41-4da2-b5e5-26cde01f7149 service neutron] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:03.381554 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c54313c5-2b65-4170-bfd6-58545998da70 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:03.382666 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c54313c5-2b65-4170-bfd6-58545998da70 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:03.382666 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c54313c5-2b65-4170-bfd6-58545998da70 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:03.382666 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c54313c5-2b65-4170-bfd6-58545998da70 admin admin] PATH_INFO: `/v3/projects/e1e846793de24f1496dadc7921868ce2` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:03.383829 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c54313c5-2b65-4170-bfd6-58545998da70 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e1e846793de24f1496dadc7921868ce2)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:03.385969 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c54313c5-2b65-4170-bfd6-58545998da70 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:03.440580 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:03.444192 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-c54313c5-2b65-4170-bfd6-58545998da70 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e1e846793de24f1496dadc7921868ce2', 'request_id': 'req-c54313c5-2b65-4170-bfd6-58545998da70'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:03.447706 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-630111bf-2347-4e6c-ab4f-5b3ecc339c6e None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:03.453651 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-c54313c5-2b65-4170-bfd6-58545998da70 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e1e846793de24f1496dadc7921868ce2', 'request_id': 'req-c54313c5-2b65-4170-bfd6-58545998da70'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:03.476044 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-c54313c5-2b65-4170-bfd6-58545998da70 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e1e846793de24f1496dadc7921868ce2', 'request_id': 'req-c54313c5-2b65-4170-bfd6-58545998da70'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:03.491313 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-c54313c5-2b65-4170-bfd6-58545998da70 admin admin] The token cache is being invalidate because project e1e846793de24f1496dadc7921868ce2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81624) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:13:03.491665 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-c54313c5-2b65-4170-bfd6-58545998da70 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:03.492541 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-c54313c5-2b65-4170-bfd6-58545998da70 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:03.494263 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1013/2056] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:13:03 2022] DELETE /identity/v3/projects/e1e846793de24f1496dadc7921868ce2 => generated 0 bytes in 146 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:13:03.498248 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c54313c5-2b65-4170-bfd6-58545998da70 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:03.499713 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-c54313c5-2b65-4170-bfd6-58545998da70 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:03.532172 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-409d7d7c-8e8a-427f-b2d1-9278febeb335 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:03.532846 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-409d7d7c-8e8a-427f-b2d1-9278febeb335 service neutron] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:03.533115 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-409d7d7c-8e8a-427f-b2d1-9278febeb335 service neutron] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:03.533993 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-409d7d7c-8e8a-427f-b2d1-9278febeb335 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:03.535979 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-409d7d7c-8e8a-427f-b2d1-9278febeb335 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:03.540662 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:03.609898 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c54313c5-2b65-4170-bfd6-58545998da70 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:03.619313 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c3a9e8a-d78d-46a5-a30c-f4fc7458af74 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:03.620335 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c3a9e8a-d78d-46a5-a30c-f4fc7458af74 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:03.620656 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c3a9e8a-d78d-46a5-a30c-f4fc7458af74 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:03.620820 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c3a9e8a-d78d-46a5-a30c-f4fc7458af74 admin admin] PATH_INFO: `/v3/users/180814bf17b14e5cbb5d39809a38621c` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:03.622064 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c3a9e8a-d78d-46a5-a30c-f4fc7458af74 admin admin] RBAC: Authorizing `identity:delete_user(user_id=180814bf17b14e5cbb5d39809a38621c)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:03.623593 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-409d7d7c-8e8a-427f-b2d1-9278febeb335 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:03.625143 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-409d7d7c-8e8a-427f-b2d1-9278febeb335 service neutron] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:03.625527 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c3a9e8a-d78d-46a5-a30c-f4fc7458af74 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:03.636556 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1044/2057] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:13:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 280 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 05:13:03.637285 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-409d7d7c-8e8a-427f-b2d1-9278febeb335 service neutron] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:03.639136 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-409d7d7c-8e8a-427f-b2d1-9278febeb335 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:03.650775 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-409d7d7c-8e8a-427f-b2d1-9278febeb335 service neutron] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:03.656620 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-595848e5-4e22-49d7-83ed-0152ec4d62f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:03.657271 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-595848e5-4e22-49d7-83ed-0152ec4d62f4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:03.657655 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-595848e5-4e22-49d7-83ed-0152ec4d62f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:03.657921 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-595848e5-4e22-49d7-83ed-0152ec4d62f4 admin admin] PATH_INFO: `/v3/projects/2a2b66d94ffc4e87804d537f1f0921b2` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:03.659811 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-595848e5-4e22-49d7-83ed-0152ec4d62f4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2a2b66d94ffc4e87804d537f1f0921b2)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:03.662055 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-595848e5-4e22-49d7-83ed-0152ec4d62f4 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:03.683381 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-8c3a9e8a-d78d-46a5-a30c-f4fc7458af74 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '180814bf17b14e5cbb5d39809a38621c'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:03.684416 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-8c3a9e8a-d78d-46a5-a30c-f4fc7458af74 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '180814bf17b14e5cbb5d39809a38621c'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:03.685622 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-8c3a9e8a-d78d-46a5-a30c-f4fc7458af74 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '180814bf17b14e5cbb5d39809a38621c'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:03.710769 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-8c3a9e8a-d78d-46a5-a30c-f4fc7458af74 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '180814bf17b14e5cbb5d39809a38621c'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:03.720711 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-8c3a9e8a-d78d-46a5-a30c-f4fc7458af74 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '180814bf17b14e5cbb5d39809a38621c'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:03.726326 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-595848e5-4e22-49d7-83ed-0152ec4d62f4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2a2b66d94ffc4e87804d537f1f0921b2', 'request_id': 'req-595848e5-4e22-49d7-83ed-0152ec4d62f4'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:03.734087 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-595848e5-4e22-49d7-83ed-0152ec4d62f4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2a2b66d94ffc4e87804d537f1f0921b2', 'request_id': 'req-595848e5-4e22-49d7-83ed-0152ec4d62f4'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:03.739214 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1014/2058] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:13:03 2022] DELETE /identity/v3/users/180814bf17b14e5cbb5d39809a38621c => generated 0 bytes in 241 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:13:03.741364 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-595848e5-4e22-49d7-83ed-0152ec4d62f4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2a2b66d94ffc4e87804d537f1f0921b2', 'request_id': 'req-595848e5-4e22-49d7-83ed-0152ec4d62f4'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:03.763937 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-595848e5-4e22-49d7-83ed-0152ec4d62f4 admin admin] The token cache is being invalidate because project 2a2b66d94ffc4e87804d537f1f0921b2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81625) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:13:03.763937 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-595848e5-4e22-49d7-83ed-0152ec4d62f4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:03.764684 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-595848e5-4e22-49d7-83ed-0152ec4d62f4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:03.767547 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1045/2059] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:13:03 2022] DELETE /identity/v3/projects/2a2b66d94ffc4e87804d537f1f0921b2 => generated 0 bytes in 130 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:13:03.772124 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c3a9e8a-d78d-46a5-a30c-f4fc7458af74 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:03.776394 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-8c3a9e8a-d78d-46a5-a30c-f4fc7458af74 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:03.880027 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c3a9e8a-d78d-46a5-a30c-f4fc7458af74 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:03.887639 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-244b5bc0-c137-48bf-a7fd-42cc6341be84 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:03.888269 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-244b5bc0-c137-48bf-a7fd-42cc6341be84 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:03.888489 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-244b5bc0-c137-48bf-a7fd-42cc6341be84 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:03.888767 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-244b5bc0-c137-48bf-a7fd-42cc6341be84 admin admin] PATH_INFO: `/v3/users/57978e4009984afaaf362947600f8497` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:03.890410 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-244b5bc0-c137-48bf-a7fd-42cc6341be84 admin admin] RBAC: Authorizing `identity:delete_user(user_id=57978e4009984afaaf362947600f8497)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:03.892918 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-244b5bc0-c137-48bf-a7fd-42cc6341be84 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:03.961657 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=83,keystone:DELETE=16,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:03.972931 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-244b5bc0-c137-48bf-a7fd-42cc6341be84 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '57978e4009984afaaf362947600f8497'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:03.973739 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-244b5bc0-c137-48bf-a7fd-42cc6341be84 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '57978e4009984afaaf362947600f8497'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:03.974669 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-244b5bc0-c137-48bf-a7fd-42cc6341be84 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '57978e4009984afaaf362947600f8497'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:03.992949 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-244b5bc0-c137-48bf-a7fd-42cc6341be84 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '57978e4009984afaaf362947600f8497'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:04.001439 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-244b5bc0-c137-48bf-a7fd-42cc6341be84 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '57978e4009984afaaf362947600f8497'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:04.028753 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1015/2060] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:13:03 2022] DELETE /identity/v3/users/57978e4009984afaaf362947600f8497 => generated 0 bytes in 257 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:13:04.075876 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-595848e5-4e22-49d7-83ed-0152ec4d62f4 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:04.077442 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-595848e5-4e22-49d7-83ed-0152ec4d62f4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:04.129687 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-244b5bc0-c137-48bf-a7fd-42cc6341be84 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:04.131505 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-244b5bc0-c137-48bf-a7fd-42cc6341be84 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:04.133841 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:04.195374 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-595848e5-4e22-49d7-83ed-0152ec4d62f4 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:04.270070 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-244b5bc0-c137-48bf-a7fd-42cc6341be84 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:04.276623 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-227dbf8c-6a18-4418-9a67-ebf38e4dac61 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:04.277368 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-227dbf8c-6a18-4418-9a67-ebf38e4dac61 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:04.277713 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-227dbf8c-6a18-4418-9a67-ebf38e4dac61 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:04.278508 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-227dbf8c-6a18-4418-9a67-ebf38e4dac61 admin admin] PATH_INFO: `/v3/projects/8dedadaa140546a3ae15a8a5c330c187` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:04.279838 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-227dbf8c-6a18-4418-9a67-ebf38e4dac61 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8dedadaa140546a3ae15a8a5c330c187)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:04.281922 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-227dbf8c-6a18-4418-9a67-ebf38e4dac61 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:04.284877 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a3c00a6-135c-4b91-be46-22485e9a36ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:04.285665 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a3c00a6-135c-4b91-be46-22485e9a36ea admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:04.285952 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a3c00a6-135c-4b91-be46-22485e9a36ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:04.286229 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a3c00a6-135c-4b91-be46-22485e9a36ea admin admin] PATH_INFO: `/v3/projects/1a8ca4ec40e84dbfa8adf3b801e31baf` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:04.287436 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a3c00a6-135c-4b91-be46-22485e9a36ea admin admin] RBAC: Authorizing `identity:delete_project(project_id=1a8ca4ec40e84dbfa8adf3b801e31baf)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:04.289358 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a3c00a6-135c-4b91-be46-22485e9a36ea admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:04.336685 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-227dbf8c-6a18-4418-9a67-ebf38e4dac61 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8dedadaa140546a3ae15a8a5c330c187', 'request_id': 'req-227dbf8c-6a18-4418-9a67-ebf38e4dac61'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:04.349132 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-1a3c00a6-135c-4b91-be46-22485e9a36ea admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1a8ca4ec40e84dbfa8adf3b801e31baf', 'request_id': 'req-1a3c00a6-135c-4b91-be46-22485e9a36ea'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:04.353858 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-227dbf8c-6a18-4418-9a67-ebf38e4dac61 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8dedadaa140546a3ae15a8a5c330c187', 'request_id': 'req-227dbf8c-6a18-4418-9a67-ebf38e4dac61'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:04.357361 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-1a3c00a6-135c-4b91-be46-22485e9a36ea admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1a8ca4ec40e84dbfa8adf3b801e31baf', 'request_id': 'req-1a3c00a6-135c-4b91-be46-22485e9a36ea'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:04.361082 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-227dbf8c-6a18-4418-9a67-ebf38e4dac61 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8dedadaa140546a3ae15a8a5c330c187', 'request_id': 'req-227dbf8c-6a18-4418-9a67-ebf38e4dac61'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:04.365974 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-1a3c00a6-135c-4b91-be46-22485e9a36ea admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1a8ca4ec40e84dbfa8adf3b801e31baf', 'request_id': 'req-1a3c00a6-135c-4b91-be46-22485e9a36ea'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:04.376342 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-227dbf8c-6a18-4418-9a67-ebf38e4dac61 admin admin] The token cache is being invalidate because project 8dedadaa140546a3ae15a8a5c330c187 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81624) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:13:04.376696 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-227dbf8c-6a18-4418-9a67-ebf38e4dac61 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:04.377725 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-1a3c00a6-135c-4b91-be46-22485e9a36ea admin admin] The token cache is being invalidate because project 1a8ca4ec40e84dbfa8adf3b801e31baf was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81625) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:13:04.377872 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-227dbf8c-6a18-4418-9a67-ebf38e4dac61 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:04.378149 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-1a3c00a6-135c-4b91-be46-22485e9a36ea admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:04.378776 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-1a3c00a6-135c-4b91-be46-22485e9a36ea admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:04.380074 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1016/2061] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:13:04 2022] DELETE /identity/v3/projects/8dedadaa140546a3ae15a8a5c330c187 => generated 0 bytes in 253 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:13:04.383124 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1046/2062] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:13:04 2022] DELETE /identity/v3/projects/1a8ca4ec40e84dbfa8adf3b801e31baf => generated 0 bytes in 305 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:13:04.384330 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-227dbf8c-6a18-4418-9a67-ebf38e4dac61 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:04.385198 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c1cced5-3e0f-46e8-afca-a68bdfa25cbd None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:04.385480 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c1cced5-3e0f-46e8-afca-a68bdfa25cbd None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:04.385718 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-227dbf8c-6a18-4418-9a67-ebf38e4dac61 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:04.385820 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c1cced5-3e0f-46e8-afca-a68bdfa25cbd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:04.407470 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-3c1cced5-3e0f-46e8-afca-a68bdfa25cbd None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:13:04.409625 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-3c1cced5-3e0f-46e8-afca-a68bdfa25cbd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:04.455540 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:DELETE=11,keystone:SELECT=87,keystone:INSERT=2 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:04.472620 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-227dbf8c-6a18-4418-9a67-ebf38e4dac61 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:04.481023 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1047/2063] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:13:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 05:13:04.481922 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c1cced5-3e0f-46e8-afca-a68bdfa25cbd None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:04.483186 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-3c1cced5-3e0f-46e8-afca-a68bdfa25cbd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:04.494927 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c1cced5-3e0f-46e8-afca-a68bdfa25cbd None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:04.499709 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-734d7813-9ec9-4719-8c45-e17b6d5f9ebd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:04.500398 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-734d7813-9ec9-4719-8c45-e17b6d5f9ebd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:04.500699 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-734d7813-9ec9-4719-8c45-e17b6d5f9ebd admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:04.501062 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-734d7813-9ec9-4719-8c45-e17b6d5f9ebd admin admin] PATH_INFO: `/v3/projects/2bccc550dc9d4980ad456b76724602ef` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:04.502220 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-734d7813-9ec9-4719-8c45-e17b6d5f9ebd admin admin] RBAC: Authorizing `identity:delete_project(project_id=2bccc550dc9d4980ad456b76724602ef)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:04.504138 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-734d7813-9ec9-4719-8c45-e17b6d5f9ebd admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:04.512792 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:04.534305 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30e6b9c3-58b8-4b30-a57f-b28509f9349a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:04.535020 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30e6b9c3-58b8-4b30-a57f-b28509f9349a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:04.535295 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30e6b9c3-58b8-4b30-a57f-b28509f9349a admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:04.535542 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30e6b9c3-58b8-4b30-a57f-b28509f9349a admin admin] PATH_INFO: `/v3/users/fdd423e995984e709a7bbde68abfc986` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:04.536687 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30e6b9c3-58b8-4b30-a57f-b28509f9349a admin admin] RBAC: Authorizing `identity:delete_user(user_id=fdd423e995984e709a7bbde68abfc986)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:04.538618 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30e6b9c3-58b8-4b30-a57f-b28509f9349a admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:04.559062 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-734d7813-9ec9-4719-8c45-e17b6d5f9ebd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2bccc550dc9d4980ad456b76724602ef', 'request_id': 'req-734d7813-9ec9-4719-8c45-e17b6d5f9ebd'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:04.567692 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-734d7813-9ec9-4719-8c45-e17b6d5f9ebd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2bccc550dc9d4980ad456b76724602ef', 'request_id': 'req-734d7813-9ec9-4719-8c45-e17b6d5f9ebd'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:04.574555 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-734d7813-9ec9-4719-8c45-e17b6d5f9ebd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2bccc550dc9d4980ad456b76724602ef', 'request_id': 'req-734d7813-9ec9-4719-8c45-e17b6d5f9ebd'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:04.585209 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-734d7813-9ec9-4719-8c45-e17b6d5f9ebd admin admin] The token cache is being invalidate because project 2bccc550dc9d4980ad456b76724602ef was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81625) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:13:04.585500 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-734d7813-9ec9-4719-8c45-e17b6d5f9ebd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:04.585835 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-30e6b9c3-58b8-4b30-a57f-b28509f9349a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fdd423e995984e709a7bbde68abfc986'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:04.586141 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-734d7813-9ec9-4719-8c45-e17b6d5f9ebd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:04.586498 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-30e6b9c3-58b8-4b30-a57f-b28509f9349a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fdd423e995984e709a7bbde68abfc986'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:04.587149 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-30e6b9c3-58b8-4b30-a57f-b28509f9349a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fdd423e995984e709a7bbde68abfc986'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:04.587651 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1048/2064] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:13:04 2022] DELETE /identity/v3/projects/2bccc550dc9d4980ad456b76724602ef => generated 0 bytes in 106 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:13:04.588808 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b43f03c-eae0-44bf-8089-6b7a40a23ced None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:04.589086 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b43f03c-eae0-44bf-8089-6b7a40a23ced None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:04.589352 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b43f03c-eae0-44bf-8089-6b7a40a23ced None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:04.606052 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-30e6b9c3-58b8-4b30-a57f-b28509f9349a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fdd423e995984e709a7bbde68abfc986'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:04.610758 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-2b43f03c-eae0-44bf-8089-6b7a40a23ced None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:13:04.612738 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-2b43f03c-eae0-44bf-8089-6b7a40a23ced None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:04.613624 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-30e6b9c3-58b8-4b30-a57f-b28509f9349a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fdd423e995984e709a7bbde68abfc986'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:04.629121 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1017/2065] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:13:04 2022] DELETE /identity/v3/users/fdd423e995984e709a7bbde68abfc986 => generated 0 bytes in 245 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:13:04.630483 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b72fbd53-16ce-40c7-8da6-9855db0a7d66 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:04.630790 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b72fbd53-16ce-40c7-8da6-9855db0a7d66 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:04.631087 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b72fbd53-16ce-40c7-8da6-9855db0a7d66 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:04.652571 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-b72fbd53-16ce-40c7-8da6-9855db0a7d66 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:13:04.653801 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-b72fbd53-16ce-40c7-8da6-9855db0a7d66 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:04.687976 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1049/2066] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:13:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:13:04.689200 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b43f03c-eae0-44bf-8089-6b7a40a23ced None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:04.699626 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b43f03c-eae0-44bf-8089-6b7a40a23ced None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:04.734795 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1018/2067] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:13:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:13:04.743945 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b3781c5-7967-4277-9f61-96f2e34a8de5 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:04.744320 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b3781c5-7967-4277-9f61-96f2e34a8de5 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:04.744664 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b3781c5-7967-4277-9f61-96f2e34a8de5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:04.766838 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-3b3781c5-7967-4277-9f61-96f2e34a8de5 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:13:04.768630 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-3b3781c5-7967-4277-9f61-96f2e34a8de5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:04.776440 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89349ed9-e3f0-443e-8daa-3c500d353397 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:04.777332 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1019/2068] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:13:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:13:04.777599 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89349ed9-e3f0-443e-8daa-3c500d353397 admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:04.778079 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89349ed9-e3f0-443e-8daa-3c500d353397 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:04.778418 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89349ed9-e3f0-443e-8daa-3c500d353397 admin admin] PATH_INFO: `/v3/domains` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:04.779108 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89349ed9-e3f0-443e-8daa-3c500d353397 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:13:04.779411 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89349ed9-e3f0-443e-8daa-3c500d353397 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:04.781727 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b3781c5-7967-4277-9f61-96f2e34a8de5 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:04.782671 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89349ed9-e3f0-443e-8daa-3c500d353397 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:04.805595 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b3781c5-7967-4277-9f61-96f2e34a8de5 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:04.811104 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f785777d-fdcf-49ba-bf61-17286d0c460e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:04.811998 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f785777d-fdcf-49ba-bf61-17286d0c460e admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:04.812918 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f785777d-fdcf-49ba-bf61-17286d0c460e admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:04.813284 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f785777d-fdcf-49ba-bf61-17286d0c460e admin admin] PATH_INFO: `/v3/domains` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:04.813848 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f785777d-fdcf-49ba-bf61-17286d0c460e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:13:04.815586 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f785777d-fdcf-49ba-bf61-17286d0c460e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:04.817728 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f785777d-fdcf-49ba-bf61-17286d0c460e admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:04.821839 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1050/2069] 10.222.0.77 () {58 vars in 1153 bytes} [Tue Aug 30 05:13:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 133 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:13:04.822755 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89349ed9-e3f0-443e-8daa-3c500d353397 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:04.827889 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:04.843602 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89349ed9-e3f0-443e-8daa-3c500d353397 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:04.849449 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1020/2070] 10.222.0.77 () {58 vars in 1153 bytes} [Tue Aug 30 05:13:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 68 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 05:13:04.850460 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f785777d-fdcf-49ba-bf61-17286d0c460e admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:04.855149 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f84e2b30-b1a8-429a-b013-f96b3daa9170 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:04.855901 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f84e2b30-b1a8-429a-b013-f96b3daa9170 admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:04.856397 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f84e2b30-b1a8-429a-b013-f96b3daa9170 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:04.856788 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f84e2b30-b1a8-429a-b013-f96b3daa9170 admin admin] PATH_INFO: `/v3/projects` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:04.857801 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f84e2b30-b1a8-429a-b013-f96b3daa9170 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:04.860187 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f84e2b30-b1a8-429a-b013-f96b3daa9170 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:04.862081 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f785777d-fdcf-49ba-bf61-17286d0c460e admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:04.867720 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e84e700b-6d56-47e9-9730-8f78dbbae5fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:04.868328 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e84e700b-6d56-47e9-9730-8f78dbbae5fb admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:04.868637 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e84e700b-6d56-47e9-9730-8f78dbbae5fb admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:04.868869 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e84e700b-6d56-47e9-9730-8f78dbbae5fb admin admin] PATH_INFO: `/v3/projects` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:04.869651 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e84e700b-6d56-47e9-9730-8f78dbbae5fb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:04.871624 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e84e700b-6d56-47e9-9730-8f78dbbae5fb admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:04.896320 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1051/2071] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:13:04 2022] POST /identity/v3/projects => generated 387 bytes in 74 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:13:04.901048 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f84e2b30-b1a8-429a-b013-f96b3daa9170 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:04.914286 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1021/2072] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:13:04 2022] POST /identity/v3/projects => generated 361 bytes in 64 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:13:04.914728 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:04.915679 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e84e700b-6d56-47e9-9730-8f78dbbae5fb admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:04.993138 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e84e700b-6d56-47e9-9730-8f78dbbae5fb admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:04.999914 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9fcb5cb-59a9-41d0-87bf-9ac6e5a4bfb8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:05.000608 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9fcb5cb-59a9-41d0-87bf-9ac6e5a4bfb8 admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:05.000956 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9fcb5cb-59a9-41d0-87bf-9ac6e5a4bfb8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:05.001296 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9fcb5cb-59a9-41d0-87bf-9ac6e5a4bfb8 admin admin] PATH_INFO: `/v3/users` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:05.001991 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9fcb5cb-59a9-41d0-87bf-9ac6e5a4bfb8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:05.004072 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9fcb5cb-59a9-41d0-87bf-9ac6e5a4bfb8 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:05.010692 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f84e2b30-b1a8-429a-b013-f96b3daa9170 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:05.017490 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3381b22a-da77-4161-8a27-52f5e4899805 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:05.018229 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3381b22a-da77-4161-8a27-52f5e4899805 admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:05.018524 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3381b22a-da77-4161-8a27-52f5e4899805 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:05.018844 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3381b22a-da77-4161-8a27-52f5e4899805 admin admin] PATH_INFO: `/v3/users` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:05.019602 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3381b22a-da77-4161-8a27-52f5e4899805 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:05.021826 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3381b22a-da77-4161-8a27-52f5e4899805 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:05.027398 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1022/2073] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:13:04 2022] POST /identity/v3/users => generated 281 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:13:05.028346 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9fcb5cb-59a9-41d0-87bf-9ac6e5a4bfb8 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:05.029972 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-c9fcb5cb-59a9-41d0-87bf-9ac6e5a4bfb8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:05.043898 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9fcb5cb-59a9-41d0-87bf-9ac6e5a4bfb8 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:05.044622 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1052/2074] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:13:04 2022] POST /identity/v3/users => generated 294 bytes in 144 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:13:05.045509 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3381b22a-da77-4161-8a27-52f5e4899805 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:05.052873 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66048b8e-df31-448b-b0e1-00d550e428ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:05.053681 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66048b8e-df31-448b-b0e1-00d550e428ba admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:05.054082 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66048b8e-df31-448b-b0e1-00d550e428ba admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:05.054247 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66048b8e-df31-448b-b0e1-00d550e428ba admin admin] PATH_INFO: `/v3/projects/fc64f952d2b14ead882fb7b39f11e304` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:05.055421 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66048b8e-df31-448b-b0e1-00d550e428ba admin admin] RBAC: Authorizing `identity:delete_project(project_id=fc64f952d2b14ead882fb7b39f11e304)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:05.055917 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3381b22a-da77-4161-8a27-52f5e4899805 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:05.057326 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66048b8e-df31-448b-b0e1-00d550e428ba admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:05.060857 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-493a9947-c54e-4ed0-960b-2b58ee39156c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:05.061625 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-493a9947-c54e-4ed0-960b-2b58ee39156c admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:05.061938 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-493a9947-c54e-4ed0-960b-2b58ee39156c admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:05.062285 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-493a9947-c54e-4ed0-960b-2b58ee39156c admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:05.063087 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-493a9947-c54e-4ed0-960b-2b58ee39156c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:05.064787 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-493a9947-c54e-4ed0-960b-2b58ee39156c admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:05.079481 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1053/2075] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:13:05 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:13:05.080465 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-493a9947-c54e-4ed0-960b-2b58ee39156c admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:05.091533 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-493a9947-c54e-4ed0-960b-2b58ee39156c admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:05.098784 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-976efeff-1fb7-44f4-a53e-3f59d6a661ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:05.099428 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-976efeff-1fb7-44f4-a53e-3f59d6a661ef admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:05.099790 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-976efeff-1fb7-44f4-a53e-3f59d6a661ef admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:05.100060 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-976efeff-1fb7-44f4-a53e-3f59d6a661ef admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:05.100698 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-976efeff-1fb7-44f4-a53e-3f59d6a661ef admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:05.102418 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-976efeff-1fb7-44f4-a53e-3f59d6a661ef admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:05.111219 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-66048b8e-df31-448b-b0e1-00d550e428ba admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fc64f952d2b14ead882fb7b39f11e304', 'request_id': 'req-66048b8e-df31-448b-b0e1-00d550e428ba'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:05.117002 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1054/2076] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:13:05 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:13:05.120962 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-976efeff-1fb7-44f4-a53e-3f59d6a661ef admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:05.121371 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-66048b8e-df31-448b-b0e1-00d550e428ba admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fc64f952d2b14ead882fb7b39f11e304', 'request_id': 'req-66048b8e-df31-448b-b0e1-00d550e428ba'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:05.129706 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-66048b8e-df31-448b-b0e1-00d550e428ba admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fc64f952d2b14ead882fb7b39f11e304', 'request_id': 'req-66048b8e-df31-448b-b0e1-00d550e428ba'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:05.137586 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-976efeff-1fb7-44f4-a53e-3f59d6a661ef admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:05.142528 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-66048b8e-df31-448b-b0e1-00d550e428ba admin admin] The token cache is being invalidate because project fc64f952d2b14ead882fb7b39f11e304 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81624) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:13:05.143445 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-66048b8e-df31-448b-b0e1-00d550e428ba admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:05.144194 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-66048b8e-df31-448b-b0e1-00d550e428ba admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:05.145989 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1023/2077] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:13:05 2022] DELETE /identity/v3/projects/fc64f952d2b14ead882fb7b39f11e304 => generated 0 bytes in 118 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:13:05.146741 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66048b8e-df31-448b-b0e1-00d550e428ba admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:05.148186 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-66048b8e-df31-448b-b0e1-00d550e428ba admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:05.201943 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=88,keystone:INSERT=5,keystone:DELETE=7 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:05.203807 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bab3d355-613d-4d48-9c46-536274b8884a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:05.204442 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bab3d355-613d-4d48-9c46-536274b8884a admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:05.204860 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bab3d355-613d-4d48-9c46-536274b8884a admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:05.205027 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bab3d355-613d-4d48-9c46-536274b8884a admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:05.205858 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bab3d355-613d-4d48-9c46-536274b8884a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:05.207360 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bab3d355-613d-4d48-9c46-536274b8884a admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:05.222948 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1055/2078] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:13:05 2022] GET /identity/v3/roles => generated 1420 bytes in 105 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:13:05.226058 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66048b8e-df31-448b-b0e1-00d550e428ba admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:05.227807 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bab3d355-613d-4d48-9c46-536274b8884a admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:05.229656 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-bab3d355-613d-4d48-9c46-536274b8884a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:05.231914 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d5d08c5-8d26-4ce9-86d4-de632edbd55e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:05.232707 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d5d08c5-8d26-4ce9-86d4-de632edbd55e admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:05.233056 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d5d08c5-8d26-4ce9-86d4-de632edbd55e admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:05.233422 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d5d08c5-8d26-4ce9-86d4-de632edbd55e admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:05.234403 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d5d08c5-8d26-4ce9-86d4-de632edbd55e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:05.237408 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d5d08c5-8d26-4ce9-86d4-de632edbd55e admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:05.244963 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bab3d355-613d-4d48-9c46-536274b8884a admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:05.251183 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a65fd90-e92e-422a-9b87-4633bde6231c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:05.251371 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1024/2079] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:13:05 2022] GET /identity/v3/roles => generated 1420 bytes in 105 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:13:05.251771 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a65fd90-e92e-422a-9b87-4633bde6231c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:05.252240 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a65fd90-e92e-422a-9b87-4633bde6231c admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:05.252655 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a65fd90-e92e-422a-9b87-4633bde6231c admin admin] PATH_INFO: `/v3/users/00713a8ed26e43958ff9012cc9afe2eb` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:05.253681 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a65fd90-e92e-422a-9b87-4633bde6231c admin admin] RBAC: Authorizing `identity:delete_user(user_id=00713a8ed26e43958ff9012cc9afe2eb)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:05.255577 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d5d08c5-8d26-4ce9-86d4-de632edbd55e admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:05.256206 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a65fd90-e92e-422a-9b87-4633bde6231c admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:05.257852 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-7d5d08c5-8d26-4ce9-86d4-de632edbd55e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:05.272953 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d5d08c5-8d26-4ce9-86d4-de632edbd55e admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:05.283175 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f69226f3-9825-4f64-a29c-f569a6dc5a53 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:05.291229 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f69226f3-9825-4f64-a29c-f569a6dc5a53 admin admin] REQUEST_METHOD: `PUT` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:05.291749 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f69226f3-9825-4f64-a29c-f569a6dc5a53 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:05.292195 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f69226f3-9825-4f64-a29c-f569a6dc5a53 admin admin] PATH_INFO: `/v3/projects/df7dad9153fe4355aaa4209c1728e909/users/36da0e7fc84149bc844f05329e14ebdd/roles/6f03d86980c34c36b44a8d6764ff4fda` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:05.312764 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f69226f3-9825-4f64-a29c-f569a6dc5a53 admin admin] RBAC: Authorizing `identity:create_grant(project_id=df7dad9153fe4355aaa4209c1728e909, user_id=36da0e7fc84149bc844f05329e14ebdd, role_id=6f03d86980c34c36b44a8d6764ff4fda)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:05.315706 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f69226f3-9825-4f64-a29c-f569a6dc5a53 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:05.324431 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-7a65fd90-e92e-422a-9b87-4633bde6231c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '00713a8ed26e43958ff9012cc9afe2eb'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:05.329214 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1025/2080] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:13:05 2022] PUT /identity/v3/projects/df7dad9153fe4355aaa4209c1728e909/users/36da0e7fc84149bc844f05329e14ebdd/roles/6f03d86980c34c36b44a8d6764ff4fda => generated 0 bytes in 74 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:13:05.330106 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f69226f3-9825-4f64-a29c-f569a6dc5a53 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:05.332512 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-7a65fd90-e92e-422a-9b87-4633bde6231c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '00713a8ed26e43958ff9012cc9afe2eb'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:05.345281 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=87,keystone:INSERT=4,keystone:DELETE=9 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:05.347789 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-7a65fd90-e92e-422a-9b87-4633bde6231c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '00713a8ed26e43958ff9012cc9afe2eb'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:05.358742 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-7a65fd90-e92e-422a-9b87-4633bde6231c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '00713a8ed26e43958ff9012cc9afe2eb'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:05.360464 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-7a65fd90-e92e-422a-9b87-4633bde6231c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '00713a8ed26e43958ff9012cc9afe2eb'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:05.368732 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1056/2081] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:13:05 2022] DELETE /identity/v3/users/00713a8ed26e43958ff9012cc9afe2eb => generated 0 bytes in 145 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:13:05.376102 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b575c4-5d91-4dff-a620-f1606aa740e4 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:05.376422 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b575c4-5d91-4dff-a620-f1606aa740e4 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:05.376820 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b575c4-5d91-4dff-a620-f1606aa740e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:05.411937 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f69226f3-9825-4f64-a29c-f569a6dc5a53 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:05.413081 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-82b575c4-5d91-4dff-a620-f1606aa740e4 None None] MFA Rules not processed for user `36da0e7fc84149bc844f05329e14ebdd`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:13:05.414306 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-82b575c4-5d91-4dff-a620-f1606aa740e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:05.512722 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a7fba90-4d5d-4eed-974a-db43a695da20 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:05.514151 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a7fba90-4d5d-4eed-974a-db43a695da20 admin admin] REQUEST_METHOD: `PUT` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:05.514743 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a7fba90-4d5d-4eed-974a-db43a695da20 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:05.515281 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a7fba90-4d5d-4eed-974a-db43a695da20 admin admin] PATH_INFO: `/v3/projects/d290987cdcab450a8ee193099b816fd3/users/ecc6d950ec1b483d95cfd1af1e34cd39/roles/6f03d86980c34c36b44a8d6764ff4fda` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:05.516187 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1057/2082] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:13:05 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:13:05.547639 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a7fba90-4d5d-4eed-974a-db43a695da20 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d290987cdcab450a8ee193099b816fd3, user_id=ecc6d950ec1b483d95cfd1af1e34cd39, role_id=6f03d86980c34c36b44a8d6764ff4fda)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:05.553244 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a7fba90-4d5d-4eed-974a-db43a695da20 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:05.564151 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82b575c4-5d91-4dff-a620-f1606aa740e4 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:05.565590 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-82b575c4-5d91-4dff-a620-f1606aa740e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:05.576386 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1026/2083] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:13:05 2022] PUT /identity/v3/projects/d290987cdcab450a8ee193099b816fd3/users/ecc6d950ec1b483d95cfd1af1e34cd39/roles/6f03d86980c34c36b44a8d6764ff4fda => generated 0 bytes in 247 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:13:05.588217 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc1941d-d1c9-46c6-8713-6d72a6678b7b None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:05.588502 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc1941d-d1c9-46c6-8713-6d72a6678b7b None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:05.588778 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc1941d-d1c9-46c6-8713-6d72a6678b7b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:05.624485 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-6cc1941d-d1c9-46c6-8713-6d72a6678b7b None None] MFA Rules not processed for user `ecc6d950ec1b483d95cfd1af1e34cd39`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:13:05.627520 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-6cc1941d-d1c9-46c6-8713-6d72a6678b7b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:05.646208 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82b575c4-5d91-4dff-a620-f1606aa740e4 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:05.676669 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:05.707300 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-969640f2-1f61-4db2-8b1a-fc3569d85ebd service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '019a79b2affe478b8a30d1c538d0415c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:05.707908 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-969640f2-1f61-4db2-8b1a-fc3569d85ebd service cinder] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:05.708408 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-969640f2-1f61-4db2-8b1a-fc3569d85ebd service cinder] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:05.708488 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-969640f2-1f61-4db2-8b1a-fc3569d85ebd service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:05.708811 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1027/2084] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:13:05 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 05:13:05.716493 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cc1941d-d1c9-46c6-8713-6d72a6678b7b None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:05.717999 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-6cc1941d-d1c9-46c6-8713-6d72a6678b7b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:05.770312 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:DELETE=1,keystone:SELECT=99 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:05.784250 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-969640f2-1f61-4db2-8b1a-fc3569d85ebd service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:05.785876 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-969640f2-1f61-4db2-8b1a-fc3569d85ebd service cinder] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:05.795969 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1058/2085] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:13:05 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 232 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 05:13:05.797492 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-969640f2-1f61-4db2-8b1a-fc3569d85ebd service cinder] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:05.798920 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-969640f2-1f61-4db2-8b1a-fc3569d85ebd service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:05.800650 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cc1941d-d1c9-46c6-8713-6d72a6678b7b None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:05.810773 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84aee644-8289-4c4a-a85a-ad551fd3ce75 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:05.810773 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84aee644-8289-4c4a-a85a-ad551fd3ce75 admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:05.810773 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84aee644-8289-4c4a-a85a-ad551fd3ce75 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:05.810773 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84aee644-8289-4c4a-a85a-ad551fd3ce75 admin admin] PATH_INFO: `/v3/projects` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:05.811156 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84aee644-8289-4c4a-a85a-ad551fd3ce75 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:05.811156 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84aee644-8289-4c4a-a85a-ad551fd3ce75 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:05.815917 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-969640f2-1f61-4db2-8b1a-fc3569d85ebd service cinder] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:05.821598 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c2b5b55-f291-49b2-abb7-25d2e24ca4bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:05.821824 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c2b5b55-f291-49b2-abb7-25d2e24ca4bf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:05.822059 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c2b5b55-f291-49b2-abb7-25d2e24ca4bf admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:05.822342 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c2b5b55-f291-49b2-abb7-25d2e24ca4bf admin admin] PATH_INFO: `/v3/projects/91ff10a8a8a44f89917cf2af46319564` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:05.823547 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c2b5b55-f291-49b2-abb7-25d2e24ca4bf admin admin] RBAC: Authorizing `identity:delete_project(project_id=91ff10a8a8a44f89917cf2af46319564)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:05.825579 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c2b5b55-f291-49b2-abb7-25d2e24ca4bf admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:05.843961 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1028/2086] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:13:05 2022] POST /identity/v3/projects => generated 381 bytes in 128 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 05:13:05.843961 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84aee644-8289-4c4a-a85a-ad551fd3ce75 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:05.843961 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-84aee644-8289-4c4a-a85a-ad551fd3ce75 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:05.875364 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-7c2b5b55-f291-49b2-abb7-25d2e24ca4bf admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '91ff10a8a8a44f89917cf2af46319564', 'request_id': 'req-7c2b5b55-f291-49b2-abb7-25d2e24ca4bf'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:05.883192 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-7c2b5b55-f291-49b2-abb7-25d2e24ca4bf admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '91ff10a8a8a44f89917cf2af46319564', 'request_id': 'req-7c2b5b55-f291-49b2-abb7-25d2e24ca4bf'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:05.889324 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-7c2b5b55-f291-49b2-abb7-25d2e24ca4bf admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '91ff10a8a8a44f89917cf2af46319564', 'request_id': 'req-7c2b5b55-f291-49b2-abb7-25d2e24ca4bf'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:05.900244 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-7c2b5b55-f291-49b2-abb7-25d2e24ca4bf admin admin] The token cache is being invalidate because project 91ff10a8a8a44f89917cf2af46319564 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81625) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:13:05.900536 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-7c2b5b55-f291-49b2-abb7-25d2e24ca4bf admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:05.901219 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-7c2b5b55-f291-49b2-abb7-25d2e24ca4bf admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:05.902734 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1059/2087] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:13:05 2022] DELETE /identity/v3/projects/91ff10a8a8a44f89917cf2af46319564 => generated 0 bytes in 106 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:13:05.903490 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c2b5b55-f291-49b2-abb7-25d2e24ca4bf admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:05.904775 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-7c2b5b55-f291-49b2-abb7-25d2e24ca4bf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:05.912347 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84aee644-8289-4c4a-a85a-ad551fd3ce75 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:05.976487 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c2b5b55-f291-49b2-abb7-25d2e24ca4bf admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:05.982985 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9ff6bb5-34b1-412f-a271-c7627695b71d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:05.983525 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9ff6bb5-34b1-412f-a271-c7627695b71d admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:05.983851 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9ff6bb5-34b1-412f-a271-c7627695b71d admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:05.984266 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9ff6bb5-34b1-412f-a271-c7627695b71d admin admin] PATH_INFO: `/v3/users` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:05.985514 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9ff6bb5-34b1-412f-a271-c7627695b71d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:05.986878 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9ff6bb5-34b1-412f-a271-c7627695b71d admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:05.987665 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60ec3e9f-93cf-475e-84ae-20c2e5144bd7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:05.988707 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60ec3e9f-93cf-475e-84ae-20c2e5144bd7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:05.989369 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60ec3e9f-93cf-475e-84ae-20c2e5144bd7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:05.989830 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60ec3e9f-93cf-475e-84ae-20c2e5144bd7 admin admin] PATH_INFO: `/v3/users/36da0e7fc84149bc844f05329e14ebdd` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:05.991678 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60ec3e9f-93cf-475e-84ae-20c2e5144bd7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=36da0e7fc84149bc844f05329e14ebdd)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:05.996638 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60ec3e9f-93cf-475e-84ae-20c2e5144bd7 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:06.016322 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1060/2088] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:13:05 2022] POST /identity/v3/users => generated 291 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:13:06.016322 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6dcea6c-2280-4a5a-862f-29c7c2006554 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:06.016322 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6dcea6c-2280-4a5a-862f-29c7c2006554 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:06.016322 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6dcea6c-2280-4a5a-862f-29c7c2006554 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:06.021077 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=98,keystone:INSERT=1,keystone:DELETE=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:06.049759 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-f6dcea6c-2280-4a5a-862f-29c7c2006554 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:13:06.051369 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-f6dcea6c-2280-4a5a-862f-29c7c2006554 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:06.060661 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1061/2089] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:13:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 47 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:13:06.061598 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6dcea6c-2280-4a5a-862f-29c7c2006554 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:06.071680 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6dcea6c-2280-4a5a-862f-29c7c2006554 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:06.074871 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-60ec3e9f-93cf-475e-84ae-20c2e5144bd7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '36da0e7fc84149bc844f05329e14ebdd'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:06.075533 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-60ec3e9f-93cf-475e-84ae-20c2e5144bd7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '36da0e7fc84149bc844f05329e14ebdd'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:06.076196 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-60ec3e9f-93cf-475e-84ae-20c2e5144bd7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '36da0e7fc84149bc844f05329e14ebdd'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:06.076592 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4654e657-d39d-4561-9507-a716aa59bce9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:06.077179 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4654e657-d39d-4561-9507-a716aa59bce9 admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:06.077456 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4654e657-d39d-4561-9507-a716aa59bce9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:06.077721 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4654e657-d39d-4561-9507-a716aa59bce9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:06.078355 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4654e657-d39d-4561-9507-a716aa59bce9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:06.079973 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4654e657-d39d-4561-9507-a716aa59bce9 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:06.088882 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-60ec3e9f-93cf-475e-84ae-20c2e5144bd7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '36da0e7fc84149bc844f05329e14ebdd'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:06.090559 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1062/2090] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:13:06 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:13:06.091882 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3afdca81-284d-4818-942b-0222aad6d7e0 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:06.092174 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3afdca81-284d-4818-942b-0222aad6d7e0 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:06.092436 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3afdca81-284d-4818-942b-0222aad6d7e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:06.097955 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-60ec3e9f-93cf-475e-84ae-20c2e5144bd7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '36da0e7fc84149bc844f05329e14ebdd'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:06.113968 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1029/2091] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:13:05 2022] DELETE /identity/v3/users/36da0e7fc84149bc844f05329e14ebdd => generated 0 bytes in 273 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:13:06.114805 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60ec3e9f-93cf-475e-84ae-20c2e5144bd7 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:06.116210 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-60ec3e9f-93cf-475e-84ae-20c2e5144bd7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:06.118305 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-3afdca81-284d-4818-942b-0222aad6d7e0 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:13:06.119526 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-3afdca81-284d-4818-942b-0222aad6d7e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:06.157246 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:06.183505 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60ec3e9f-93cf-475e-84ae-20c2e5144bd7 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:06.185054 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1063/2092] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:13:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:13:06.186274 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91dcae58-6a11-41ca-a38a-19c5f9440ffe None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:06.186562 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91dcae58-6a11-41ca-a38a-19c5f9440ffe None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:06.186857 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91dcae58-6a11-41ca-a38a-19c5f9440ffe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:06.188811 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75bf1e18-3bf4-4ba3-b23e-da700c40f8e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:06.189416 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75bf1e18-3bf4-4ba3-b23e-da700c40f8e5 admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:06.190278 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75bf1e18-3bf4-4ba3-b23e-da700c40f8e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:06.190580 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75bf1e18-3bf4-4ba3-b23e-da700c40f8e5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:06.191347 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75bf1e18-3bf4-4ba3-b23e-da700c40f8e5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:06.193022 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75bf1e18-3bf4-4ba3-b23e-da700c40f8e5 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:06.207858 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1030/2093] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:13:06 2022] GET /identity/v3/roles => generated 1420 bytes in 93 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:13:06.208067 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-91dcae58-6a11-41ca-a38a-19c5f9440ffe None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:13:06.209138 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75bf1e18-3bf4-4ba3-b23e-da700c40f8e5 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:06.209270 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-91dcae58-6a11-41ca-a38a-19c5f9440ffe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:06.220182 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1064/2094] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:13:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:13:06.220182 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91dcae58-6a11-41ca-a38a-19c5f9440ffe None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:06.220431 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75bf1e18-3bf4-4ba3-b23e-da700c40f8e5 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:06.225906 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86c936e4-9894-4c00-8957-644303c13c87 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:06.226834 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86c936e4-9894-4c00-8957-644303c13c87 admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:06.227074 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86c936e4-9894-4c00-8957-644303c13c87 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:06.227346 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86c936e4-9894-4c00-8957-644303c13c87 admin admin] PATH_INFO: `/v3/domains` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:06.227457 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91dcae58-6a11-41ca-a38a-19c5f9440ffe None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:06.227988 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86c936e4-9894-4c00-8957-644303c13c87 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:13:06.228289 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86c936e4-9894-4c00-8957-644303c13c87 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:06.230178 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86c936e4-9894-4c00-8957-644303c13c87 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:06.232550 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df0f817e-300f-4e44-8027-abbbb86456fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:06.233683 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df0f817e-300f-4e44-8027-abbbb86456fa admin admin] REQUEST_METHOD: `PUT` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:06.233959 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df0f817e-300f-4e44-8027-abbbb86456fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:06.234255 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df0f817e-300f-4e44-8027-abbbb86456fa admin admin] PATH_INFO: `/v3/projects/0bbdf5ad881a488eb068c16d5bde6646/users/f8ffc73545db4ee4af3cddc8bfa1b781/roles/6f03d86980c34c36b44a8d6764ff4fda` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:06.243356 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1031/2095] 10.222.0.77 () {58 vars in 1153 bytes} [Tue Aug 30 05:13:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:13:06.244157 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86c936e4-9894-4c00-8957-644303c13c87 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:06.245843 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-86c936e4-9894-4c00-8957-644303c13c87 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:06.253023 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df0f817e-300f-4e44-8027-abbbb86456fa admin admin] RBAC: Authorizing `identity:create_grant(project_id=0bbdf5ad881a488eb068c16d5bde6646, user_id=f8ffc73545db4ee4af3cddc8bfa1b781, role_id=6f03d86980c34c36b44a8d6764ff4fda)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:06.255931 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df0f817e-300f-4e44-8027-abbbb86456fa admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:06.269704 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1065/2096] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:13:06 2022] PUT /identity/v3/projects/0bbdf5ad881a488eb068c16d5bde6646/users/f8ffc73545db4ee4af3cddc8bfa1b781/roles/6f03d86980c34c36b44a8d6764ff4fda => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:13:06.270629 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df0f817e-300f-4e44-8027-abbbb86456fa admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:06.329555 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86c936e4-9894-4c00-8957-644303c13c87 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:06.358578 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df0f817e-300f-4e44-8027-abbbb86456fa admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:06.365109 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d946bad-292d-4759-a827-45ec6d654193 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:06.366128 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d946bad-292d-4759-a827-45ec6d654193 admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:06.366608 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d946bad-292d-4759-a827-45ec6d654193 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:06.367024 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d946bad-292d-4759-a827-45ec6d654193 admin admin] PATH_INFO: `/v3/projects` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:06.367692 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d946bad-292d-4759-a827-45ec6d654193 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:06.369454 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:06.369713 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d946bad-292d-4759-a827-45ec6d654193 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:06.396218 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94f8a9fc-eac2-49d7-adc9-dc7657f144ef service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:06.396835 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94f8a9fc-eac2-49d7-adc9-dc7657f144ef service neutron] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:06.397117 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94f8a9fc-eac2-49d7-adc9-dc7657f144ef service neutron] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:06.397386 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94f8a9fc-eac2-49d7-adc9-dc7657f144ef service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:06.402395 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1066/2097] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:13:06 2022] POST /identity/v3/projects => generated 369 bytes in 132 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:13:06.403539 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2033029f-acc6-4b1f-bdd2-b6ecfbeb4d87 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:06.403986 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2033029f-acc6-4b1f-bdd2-b6ecfbeb4d87 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:06.404318 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2033029f-acc6-4b1f-bdd2-b6ecfbeb4d87 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:06.411581 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94f8a9fc-eac2-49d7-adc9-dc7657f144ef service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:06.413038 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94f8a9fc-eac2-49d7-adc9-dc7657f144ef service neutron] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:06.423382 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1032/2098] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:13:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:13:06.425106 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94f8a9fc-eac2-49d7-adc9-dc7657f144ef service neutron] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:06.435081 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-2033029f-acc6-4b1f-bdd2-b6ecfbeb4d87 None None] MFA Rules not processed for user `f8ffc73545db4ee4af3cddc8bfa1b781`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:13:06.435445 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94f8a9fc-eac2-49d7-adc9-dc7657f144ef service neutron] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:06.437250 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-2033029f-acc6-4b1f-bdd2-b6ecfbeb4d87 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:06.511179 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1067/2099] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:13:06 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:13:06.515301 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2033029f-acc6-4b1f-bdd2-b6ecfbeb4d87 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:06.536940 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78027145-9417-4d01-8b52-1cd4a7655f02 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:06.537557 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78027145-9417-4d01-8b52-1cd4a7655f02 admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:06.537842 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78027145-9417-4d01-8b52-1cd4a7655f02 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:06.538117 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78027145-9417-4d01-8b52-1cd4a7655f02 admin admin] PATH_INFO: `/v3/users` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:06.539019 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78027145-9417-4d01-8b52-1cd4a7655f02 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:06.541906 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78027145-9417-4d01-8b52-1cd4a7655f02 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:06.546278 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:06.580040 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1033/2100] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:13:06 2022] POST /identity/v3/users => generated 285 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:13:06.581327 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78027145-9417-4d01-8b52-1cd4a7655f02 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:06.603103 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78027145-9417-4d01-8b52-1cd4a7655f02 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:06.612634 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f046ff7-e930-483b-abfe-ba149495a2a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:06.613715 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f046ff7-e930-483b-abfe-ba149495a2a9 admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:06.614070 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f046ff7-e930-483b-abfe-ba149495a2a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:06.614395 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f046ff7-e930-483b-abfe-ba149495a2a9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:06.614526 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2033029f-acc6-4b1f-bdd2-b6ecfbeb4d87 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:06.615251 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f046ff7-e930-483b-abfe-ba149495a2a9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:06.617609 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f046ff7-e930-483b-abfe-ba149495a2a9 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:06.620000 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15652c0c-96c6-438e-8881-e0b3688bfcab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:06.621247 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15652c0c-96c6-438e-8881-e0b3688bfcab admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:06.621865 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15652c0c-96c6-438e-8881-e0b3688bfcab admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:06.622147 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15652c0c-96c6-438e-8881-e0b3688bfcab admin admin] PATH_INFO: `/v3/projects` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:06.622893 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15652c0c-96c6-438e-8881-e0b3688bfcab admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:06.625331 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15652c0c-96c6-438e-8881-e0b3688bfcab admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:06.633924 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1034/2101] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:13:06 2022] GET /identity/v3/roles => generated 1420 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:13:06.637578 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f046ff7-e930-483b-abfe-ba149495a2a9 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:06.661783 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f046ff7-e930-483b-abfe-ba149495a2a9 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:06.667681 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4edbffae-8309-45bb-ad5a-9087c4d5fbf1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:06.668300 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4edbffae-8309-45bb-ad5a-9087c4d5fbf1 admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:06.668566 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4edbffae-8309-45bb-ad5a-9087c4d5fbf1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:06.668839 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4edbffae-8309-45bb-ad5a-9087c4d5fbf1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:06.669477 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4edbffae-8309-45bb-ad5a-9087c4d5fbf1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:06.671767 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4edbffae-8309-45bb-ad5a-9087c4d5fbf1 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:06.678238 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1068/2102] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:13:06 2022] POST /identity/v3/projects => generated 385 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 05:13:06.683067 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15652c0c-96c6-438e-8881-e0b3688bfcab admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:06.688094 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1035/2103] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:13:06 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:13:06.694381 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4edbffae-8309-45bb-ad5a-9087c4d5fbf1 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:06.720439 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4edbffae-8309-45bb-ad5a-9087c4d5fbf1 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:06.790971 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15652c0c-96c6-438e-8881-e0b3688bfcab admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:06.795764 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e94a267f-3045-4ab5-b78e-97b85e7288d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:06.796368 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e94a267f-3045-4ab5-b78e-97b85e7288d0 admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:06.796645 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e94a267f-3045-4ab5-b78e-97b85e7288d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:06.796916 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e94a267f-3045-4ab5-b78e-97b85e7288d0 admin admin] PATH_INFO: `/v3/users` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:06.797547 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e94a267f-3045-4ab5-b78e-97b85e7288d0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:06.799597 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e94a267f-3045-4ab5-b78e-97b85e7288d0 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:06.812141 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ff47555-a74e-436c-b005-62a7405f5aee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:06.812911 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff47555-a74e-436c-b005-62a7405f5aee admin admin] REQUEST_METHOD: `PUT` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:06.813224 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff47555-a74e-436c-b005-62a7405f5aee admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:06.813502 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff47555-a74e-436c-b005-62a7405f5aee admin admin] PATH_INFO: `/v3/projects/ea69acda2e50431380c9df87e247cb2e/users/4032879b46694b958677eb2465e4e5f0/roles/6f03d86980c34c36b44a8d6764ff4fda` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:06.825632 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1069/2104] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:13:06 2022] POST /identity/v3/users => generated 299 bytes in 143 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:13:06.829460 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e94a267f-3045-4ab5-b78e-97b85e7288d0 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:06.832910 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ff47555-a74e-436c-b005-62a7405f5aee admin admin] RBAC: Authorizing `identity:create_grant(project_id=ea69acda2e50431380c9df87e247cb2e, user_id=4032879b46694b958677eb2465e4e5f0, role_id=6f03d86980c34c36b44a8d6764ff4fda)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:06.835735 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ff47555-a74e-436c-b005-62a7405f5aee admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:06.841521 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e94a267f-3045-4ab5-b78e-97b85e7288d0 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:06.847280 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e52625fd-584f-4820-a16a-c190a8f752d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:06.847882 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e52625fd-584f-4820-a16a-c190a8f752d9 admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:06.848159 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e52625fd-584f-4820-a16a-c190a8f752d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:06.848428 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e52625fd-584f-4820-a16a-c190a8f752d9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:06.849068 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e52625fd-584f-4820-a16a-c190a8f752d9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:06.850693 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e52625fd-584f-4820-a16a-c190a8f752d9 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:06.861147 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1036/2105] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:13:06 2022] PUT /identity/v3/projects/ea69acda2e50431380c9df87e247cb2e/users/4032879b46694b958677eb2465e4e5f0/roles/6f03d86980c34c36b44a8d6764ff4fda => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:13:06.862920 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1070/2106] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:13:06 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 05:13:06.868151 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ff47555-a74e-436c-b005-62a7405f5aee admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:06.912920 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:06.948155 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eb3d333-1c44-4fc4-ab43-09df1fd15bf0 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:06.950646 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eb3d333-1c44-4fc4-ab43-09df1fd15bf0 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:06.953240 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eb3d333-1c44-4fc4-ab43-09df1fd15bf0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:06.957713 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ff47555-a74e-436c-b005-62a7405f5aee admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:06.971883 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b3eb9e5-2c44-408e-965f-cd9c214cf3d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:06.972680 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b3eb9e5-2c44-408e-965f-cd9c214cf3d8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:06.972992 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b3eb9e5-2c44-408e-965f-cd9c214cf3d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:06.973400 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b3eb9e5-2c44-408e-965f-cd9c214cf3d8 admin admin] PATH_INFO: `/v3/projects/bd0607128d5f4ece9192e93377e6f729/users/80d4100aee7a4b86af5ae4f2baae4679/roles/9d9d5d2c270146488249b7ed943280db` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:06.988380 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-4eb3d333-1c44-4fc4-ab43-09df1fd15bf0 None None] MFA Rules not processed for user `4032879b46694b958677eb2465e4e5f0`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:13:06.989563 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-4eb3d333-1c44-4fc4-ab43-09df1fd15bf0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:06.991825 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b3eb9e5-2c44-408e-965f-cd9c214cf3d8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bd0607128d5f4ece9192e93377e6f729, user_id=80d4100aee7a4b86af5ae4f2baae4679, role_id=9d9d5d2c270146488249b7ed943280db)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:06.998536 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b3eb9e5-2c44-408e-965f-cd9c214cf3d8 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:07.014027 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1037/2107] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:13:06 2022] PUT /identity/v3/projects/bd0607128d5f4ece9192e93377e6f729/users/80d4100aee7a4b86af5ae4f2baae4679/roles/9d9d5d2c270146488249b7ed943280db => generated 0 bytes in 146 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:13:07.015075 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b3eb9e5-2c44-408e-965f-cd9c214cf3d8 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:07.016751 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-9b3eb9e5-2c44-408e-965f-cd9c214cf3d8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:07.064454 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1071/2108] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:13:06 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 05:13:07.066064 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eb3d333-1c44-4fc4-ab43-09df1fd15bf0 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:07.077667 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eb3d333-1c44-4fc4-ab43-09df1fd15bf0 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:07.091746 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:07.093244 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b3eb9e5-2c44-408e-965f-cd9c214cf3d8 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:07.099882 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a13b7816-7d05-4302-b06e-9b7ed752c45f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:07.100669 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a13b7816-7d05-4302-b06e-9b7ed752c45f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:07.101131 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a13b7816-7d05-4302-b06e-9b7ed752c45f admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:07.101411 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a13b7816-7d05-4302-b06e-9b7ed752c45f admin admin] PATH_INFO: `/v3/projects/df7dad9153fe4355aaa4209c1728e909` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:07.102682 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a13b7816-7d05-4302-b06e-9b7ed752c45f admin admin] RBAC: Authorizing `identity:delete_project(project_id=df7dad9153fe4355aaa4209c1728e909)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:07.104703 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a13b7816-7d05-4302-b06e-9b7ed752c45f admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:07.142862 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6e68d16-75a8-42c2-be4b-1d714eed4b9d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:07.143645 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6e68d16-75a8-42c2-be4b-1d714eed4b9d admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:07.143929 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6e68d16-75a8-42c2-be4b-1d714eed4b9d admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:07.144249 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6e68d16-75a8-42c2-be4b-1d714eed4b9d admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:07.144911 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6e68d16-75a8-42c2-be4b-1d714eed4b9d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:07.146538 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6e68d16-75a8-42c2-be4b-1d714eed4b9d admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:07.158107 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1072/2109] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:13:07 2022] GET /identity/v3/roles => generated 1420 bytes in 93 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:13:07.159167 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6e68d16-75a8-42c2-be4b-1d714eed4b9d admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:07.160859 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-e6e68d16-75a8-42c2-be4b-1d714eed4b9d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:07.165256 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-a13b7816-7d05-4302-b06e-9b7ed752c45f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'df7dad9153fe4355aaa4209c1728e909', 'request_id': 'req-a13b7816-7d05-4302-b06e-9b7ed752c45f'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:07.175287 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-a13b7816-7d05-4302-b06e-9b7ed752c45f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'df7dad9153fe4355aaa4209c1728e909', 'request_id': 'req-a13b7816-7d05-4302-b06e-9b7ed752c45f'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:07.181729 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-a13b7816-7d05-4302-b06e-9b7ed752c45f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'df7dad9153fe4355aaa4209c1728e909', 'request_id': 'req-a13b7816-7d05-4302-b06e-9b7ed752c45f'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:07.192479 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-a13b7816-7d05-4302-b06e-9b7ed752c45f admin admin] The token cache is being invalidate because project df7dad9153fe4355aaa4209c1728e909 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81624) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:13:07.192763 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-a13b7816-7d05-4302-b06e-9b7ed752c45f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:07.193390 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-a13b7816-7d05-4302-b06e-9b7ed752c45f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:07.194950 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1038/2110] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:13:07 2022] DELETE /identity/v3/projects/df7dad9153fe4355aaa4209c1728e909 => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:13:07.195668 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a13b7816-7d05-4302-b06e-9b7ed752c45f admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:07.196978 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-a13b7816-7d05-4302-b06e-9b7ed752c45f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:07.222863 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6e68d16-75a8-42c2-be4b-1d714eed4b9d admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:07.240611 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:07.272016 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a13b7816-7d05-4302-b06e-9b7ed752c45f admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:07.276720 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3ddf9a8-b636-447c-a539-0f56daafaf7f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:07.276909 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-095ef2ec-aeb9-4694-bc8f-276b8d90df29 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a002ac0bf240a891ac7304178d3f0e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:07.277382 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3ddf9a8-b636-447c-a539-0f56daafaf7f admin admin] REQUEST_METHOD: `PUT` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:07.277485 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-095ef2ec-aeb9-4694-bc8f-276b8d90df29 service glance] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:07.277660 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3ddf9a8-b636-447c-a539-0f56daafaf7f admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:07.277763 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-095ef2ec-aeb9-4694-bc8f-276b8d90df29 service glance] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:07.277928 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-095ef2ec-aeb9-4694-bc8f-276b8d90df29 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:07.278018 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3ddf9a8-b636-447c-a539-0f56daafaf7f admin admin] PATH_INFO: `/v3/projects/bd0607128d5f4ece9192e93377e6f729/users/80d4100aee7a4b86af5ae4f2baae4679/roles/9d9d5d2c270146488249b7ed943280db` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:07.279858 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3ddf9a8-b636-447c-a539-0f56daafaf7f admin admin] RBAC: Authorizing `identity:create_grant(project_id=bd0607128d5f4ece9192e93377e6f729, user_id=80d4100aee7a4b86af5ae4f2baae4679, role_id=9d9d5d2c270146488249b7ed943280db)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:07.280253 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-095ef2ec-aeb9-4694-bc8f-276b8d90df29 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:07.282669 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3ddf9a8-b636-447c-a539-0f56daafaf7f admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:07.295194 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1039/2111] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:13:07 2022] PUT /identity/v3/projects/bd0607128d5f4ece9192e93377e6f729/users/80d4100aee7a4b86af5ae4f2baae4679/roles/9d9d5d2c270146488249b7ed943280db => generated 0 bytes in 100 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:13:07.296408 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2716aadd-8de2-409b-811d-0f6fa6bd651a None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:07.296689 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2716aadd-8de2-409b-811d-0f6fa6bd651a None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:07.296960 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2716aadd-8de2-409b-811d-0f6fa6bd651a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:07.322314 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-2716aadd-8de2-409b-811d-0f6fa6bd651a None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:13:07.323532 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-2716aadd-8de2-409b-811d-0f6fa6bd651a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:07.342382 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-095ef2ec-aeb9-4694-bc8f-276b8d90df29 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:07.343851 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-095ef2ec-aeb9-4694-bc8f-276b8d90df29 service glance] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:07.353890 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1073/2112] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:13:07 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 195 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:13:07.355347 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ea8ee48-847a-4f8e-ab2f-dc71c96c07c1 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:07.355639 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ea8ee48-847a-4f8e-ab2f-dc71c96c07c1 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:07.355918 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ea8ee48-847a-4f8e-ab2f-dc71c96c07c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:07.390142 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1040/2113] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:13:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:13:07.390784 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-5ea8ee48-847a-4f8e-ab2f-dc71c96c07c1 None None] MFA Rules not processed for user `80d4100aee7a4b86af5ae4f2baae4679`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:13:07.390784 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-5ea8ee48-847a-4f8e-ab2f-dc71c96c07c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:07.391733 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c50773f-669a-4672-9bd5-573bb08f3596 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:07.392024 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c50773f-669a-4672-9bd5-573bb08f3596 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:07.393116 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c50773f-669a-4672-9bd5-573bb08f3596 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:07.403754 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:07.413220 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-9c50773f-669a-4672-9bd5-573bb08f3596 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:13:07.414433 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-9c50773f-669a-4672-9bd5-573bb08f3596 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:07.422749 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1041/2114] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:13:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 05:13:07.427412 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c50773f-669a-4672-9bd5-573bb08f3596 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:07.436321 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c50773f-669a-4672-9bd5-573bb08f3596 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:07.441269 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2435a15-fe2e-44d0-baaa-163db8ef4141 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:07.441941 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2435a15-fe2e-44d0-baaa-163db8ef4141 admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:07.442281 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2435a15-fe2e-44d0-baaa-163db8ef4141 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:07.442636 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2435a15-fe2e-44d0-baaa-163db8ef4141 admin admin] PATH_INFO: `/v3/domains` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:07.443500 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2435a15-fe2e-44d0-baaa-163db8ef4141 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:13:07.443866 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2435a15-fe2e-44d0-baaa-163db8ef4141 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:07.445523 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2435a15-fe2e-44d0-baaa-163db8ef4141 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:07.469342 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1042/2115] 10.222.0.77 () {58 vars in 1153 bytes} [Tue Aug 30 05:13:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 05:13:07.474071 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2435a15-fe2e-44d0-baaa-163db8ef4141 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:07.475200 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1074/2116] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:13:07 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:13:07.484883 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2435a15-fe2e-44d0-baaa-163db8ef4141 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:07.486132 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ea8ee48-847a-4f8e-ab2f-dc71c96c07c1 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:07.488366 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-5ea8ee48-847a-4f8e-ab2f-dc71c96c07c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:07.491831 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11ce4712-6560-4ff6-abb9-6e6563bdfbae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:07.492381 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ce4712-6560-4ff6-abb9-6e6563bdfbae admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:07.492663 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ce4712-6560-4ff6-abb9-6e6563bdfbae admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:07.492939 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ce4712-6560-4ff6-abb9-6e6563bdfbae admin admin] PATH_INFO: `/v3/projects` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:07.493563 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11ce4712-6560-4ff6-abb9-6e6563bdfbae admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:07.495520 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11ce4712-6560-4ff6-abb9-6e6563bdfbae admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:07.531672 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1043/2117] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:13:07 2022] POST /identity/v3/projects => generated 373 bytes in 58 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 05:13:07.536136 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11ce4712-6560-4ff6-abb9-6e6563bdfbae admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:07.559920 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ea8ee48-847a-4f8e-ab2f-dc71c96c07c1 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:07.613177 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11ce4712-6560-4ff6-abb9-6e6563bdfbae admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:07.626275 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2577eb6a-36d6-4bab-b6d6-5b8bb0fd4c06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:07.626932 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2577eb6a-36d6-4bab-b6d6-5b8bb0fd4c06 admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:07.627255 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2577eb6a-36d6-4bab-b6d6-5b8bb0fd4c06 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:07.627527 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2577eb6a-36d6-4bab-b6d6-5b8bb0fd4c06 admin admin] PATH_INFO: `/v3/users` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:07.628166 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2577eb6a-36d6-4bab-b6d6-5b8bb0fd4c06 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:07.629343 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-202f7f15-84e4-483b-b60b-02427937a6e7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '019a79b2affe478b8a30d1c538d0415c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:07.629952 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-202f7f15-84e4-483b-b60b-02427937a6e7 service cinder] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:07.630169 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2577eb6a-36d6-4bab-b6d6-5b8bb0fd4c06 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:07.630286 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-202f7f15-84e4-483b-b60b-02427937a6e7 service cinder] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:07.630489 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-202f7f15-84e4-483b-b60b-02427937a6e7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:07.657557 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1044/2118] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:13:07 2022] POST /identity/v3/users => generated 287 bytes in 119 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:13:07.659431 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2577eb6a-36d6-4bab-b6d6-5b8bb0fd4c06 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:07.673266 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2577eb6a-36d6-4bab-b6d6-5b8bb0fd4c06 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:07.678846 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e06e20-9965-4200-9601-2a5860d30784 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:07.679563 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e06e20-9965-4200-9601-2a5860d30784 admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:07.679911 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e06e20-9965-4200-9601-2a5860d30784 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:07.680249 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e06e20-9965-4200-9601-2a5860d30784 admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:07.680964 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66e06e20-9965-4200-9601-2a5860d30784 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:07.682805 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66e06e20-9965-4200-9601-2a5860d30784 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:07.696262 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1045/2119] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:13:07 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 05:13:07.704035 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e06e20-9965-4200-9601-2a5860d30784 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:07.714206 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-202f7f15-84e4-483b-b60b-02427937a6e7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:07.716049 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-202f7f15-84e4-483b-b60b-02427937a6e7 service cinder] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:07.718132 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e06e20-9965-4200-9601-2a5860d30784 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:07.725849 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6451fb2-f27d-4825-a7c7-6419bcb1178c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:07.726443 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6451fb2-f27d-4825-a7c7-6419bcb1178c admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:07.727336 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6451fb2-f27d-4825-a7c7-6419bcb1178c admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:07.727779 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6451fb2-f27d-4825-a7c7-6419bcb1178c admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:07.728900 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6451fb2-f27d-4825-a7c7-6419bcb1178c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:07.730127 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1075/2120] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:13:07 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 244 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:13:07.731456 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6451fb2-f27d-4825-a7c7-6419bcb1178c admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:07.735360 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:07.747903 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1046/2121] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:13:07 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 05:13:07.752406 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-202f7f15-84e4-483b-b60b-02427937a6e7 service cinder] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:07.764150 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-202f7f15-84e4-483b-b60b-02427937a6e7 service cinder] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:07.771067 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79dc0589-5ad4-4d1d-813f-f18fc433df61 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:07.771905 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79dc0589-5ad4-4d1d-813f-f18fc433df61 admin admin] REQUEST_METHOD: `PUT` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:07.772191 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79dc0589-5ad4-4d1d-813f-f18fc433df61 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:07.772465 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79dc0589-5ad4-4d1d-813f-f18fc433df61 admin admin] PATH_INFO: `/v3/projects/9cea756f95e843b5868e53f6446d648a/users/b50d70f00f33494e81d4aac49df96dc4/roles/6f03d86980c34c36b44a8d6764ff4fda` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:07.790129 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:07.795254 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79dc0589-5ad4-4d1d-813f-f18fc433df61 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9cea756f95e843b5868e53f6446d648a, user_id=b50d70f00f33494e81d4aac49df96dc4, role_id=6f03d86980c34c36b44a8d6764ff4fda)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:07.798643 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79dc0589-5ad4-4d1d-813f-f18fc433df61 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:07.819615 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1076/2122] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:13:07 2022] PUT /identity/v3/projects/9cea756f95e843b5868e53f6446d648a/users/b50d70f00f33494e81d4aac49df96dc4/roles/6f03d86980c34c36b44a8d6764ff4fda => generated 0 bytes in 68 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 05:13:07.824754 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff77d6a9-2ae9-459a-bcfb-e12855f02f2b None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:07.825253 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff77d6a9-2ae9-459a-bcfb-e12855f02f2b None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:07.825620 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff77d6a9-2ae9-459a-bcfb-e12855f02f2b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:07.848547 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-ff77d6a9-2ae9-459a-bcfb-e12855f02f2b None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:13:07.849860 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-ff77d6a9-2ae9-459a-bcfb-e12855f02f2b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:07.950162 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1047/2123] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:13:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:13:07.961404 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79dc0589-5ad4-4d1d-813f-f18fc433df61 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:07.963588 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-79dc0589-5ad4-4d1d-813f-f18fc433df61 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:08.041126 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79dc0589-5ad4-4d1d-813f-f18fc433df61 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:08.045563 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-534b8da1-4e75-4c8d-ac05-9b41ee9fe418 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:08.046488 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-534b8da1-4e75-4c8d-ac05-9b41ee9fe418 service neutron] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:08.046747 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-534b8da1-4e75-4c8d-ac05-9b41ee9fe418 service neutron] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:08.047274 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-534b8da1-4e75-4c8d-ac05-9b41ee9fe418 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:08.057122 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-534b8da1-4e75-4c8d-ac05-9b41ee9fe418 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:08.058648 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-534b8da1-4e75-4c8d-ac05-9b41ee9fe418 service neutron] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:08.071360 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1077/2124] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:13:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:13:08.447969 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff77d6a9-2ae9-459a-bcfb-e12855f02f2b None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:08.526690 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff77d6a9-2ae9-459a-bcfb-e12855f02f2b None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:08.530690 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff69e9d1-37ae-45a9-8f1f-0508670f5736 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '019a79b2affe478b8a30d1c538d0415c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:08.531602 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff69e9d1-37ae-45a9-8f1f-0508670f5736 service cinder] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:08.532031 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff69e9d1-37ae-45a9-8f1f-0508670f5736 service cinder] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:08.532472 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff69e9d1-37ae-45a9-8f1f-0508670f5736 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:08.535281 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-ff69e9d1-37ae-45a9-8f1f-0508670f5736 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:08.614306 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff69e9d1-37ae-45a9-8f1f-0508670f5736 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:08.615812 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff69e9d1-37ae-45a9-8f1f-0508670f5736 service cinder] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:08.625044 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1048/2125] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:13:08 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 178 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:13:08.835596 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-594e10bd-daec-4600-a101-349df47d6a57 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:08.836106 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-594e10bd-daec-4600-a101-349df47d6a57 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:08.836228 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-594e10bd-daec-4600-a101-349df47d6a57 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:08.873893 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-594e10bd-daec-4600-a101-349df47d6a57 None None] MFA Rules not processed for user `b50d70f00f33494e81d4aac49df96dc4`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:13:08.875410 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-594e10bd-daec-4600-a101-349df47d6a57 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:08.993819 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1078/2126] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:13:08 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:13:09.014217 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff69e9d1-37ae-45a9-8f1f-0508670f5736 service cinder] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:09.038189 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff69e9d1-37ae-45a9-8f1f-0508670f5736 service cinder] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:09.051334 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d829db0e-d934-4a28-8089-ed2c52056d94 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '019a79b2affe478b8a30d1c538d0415c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:09.052191 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d829db0e-d934-4a28-8089-ed2c52056d94 service cinder] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:09.052442 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d829db0e-d934-4a28-8089-ed2c52056d94 service cinder] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:09.052780 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d829db0e-d934-4a28-8089-ed2c52056d94 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:09.064042 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d829db0e-d934-4a28-8089-ed2c52056d94 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:09.066252 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d829db0e-d934-4a28-8089-ed2c52056d94 service cinder] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:09.080200 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1049/2127] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:13:09 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 66 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:13:10.511726 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c333e401-67af-4946-9077-30b2193e0192 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:10.512228 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c333e401-67af-4946-9077-30b2193e0192 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:10.512320 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c333e401-67af-4946-9077-30b2193e0192 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:10.536592 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-c333e401-67af-4946-9077-30b2193e0192 None None] MFA Rules not processed for user `b50d70f00f33494e81d4aac49df96dc4`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:13:10.538145 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-c333e401-67af-4946-9077-30b2193e0192 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:10.546085 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1079/2128] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:13:10 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:13:12.801731 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d829db0e-d934-4a28-8089-ed2c52056d94 service cinder] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:12.810196 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d829db0e-d934-4a28-8089-ed2c52056d94 service cinder] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:12.813723 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeefd5a9-b9ab-4501-b184-7bfde7a841ff service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '019a79b2affe478b8a30d1c538d0415c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:12.814323 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeefd5a9-b9ab-4501-b184-7bfde7a841ff service cinder] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:12.814598 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeefd5a9-b9ab-4501-b184-7bfde7a841ff service cinder] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:12.815415 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeefd5a9-b9ab-4501-b184-7bfde7a841ff service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:12.817079 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-eeefd5a9-b9ab-4501-b184-7bfde7a841ff service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:12.859999 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:12.883427 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeefd5a9-b9ab-4501-b184-7bfde7a841ff service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:12.884944 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeefd5a9-b9ab-4501-b184-7bfde7a841ff service cinder] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:12.893920 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1050/2129] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:13:12 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:13:15.234227 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c333e401-67af-4946-9077-30b2193e0192 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:15.256112 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c333e401-67af-4946-9077-30b2193e0192 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:15.263424 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-984dc16c-c244-4c45-95f5-d451f7b84ee5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:15.264320 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-984dc16c-c244-4c45-95f5-d451f7b84ee5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:15.264672 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-984dc16c-c244-4c45-95f5-d451f7b84ee5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:15.265015 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-984dc16c-c244-4c45-95f5-d451f7b84ee5 admin admin] PATH_INFO: `/v3/users/ecc6d950ec1b483d95cfd1af1e34cd39` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:15.271862 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-984dc16c-c244-4c45-95f5-d451f7b84ee5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ecc6d950ec1b483d95cfd1af1e34cd39)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:15.274032 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-984dc16c-c244-4c45-95f5-d451f7b84ee5 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:15.328764 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-984dc16c-c244-4c45-95f5-d451f7b84ee5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ecc6d950ec1b483d95cfd1af1e34cd39'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:15.337645 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-984dc16c-c244-4c45-95f5-d451f7b84ee5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ecc6d950ec1b483d95cfd1af1e34cd39'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:15.352028 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-984dc16c-c244-4c45-95f5-d451f7b84ee5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ecc6d950ec1b483d95cfd1af1e34cd39'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:15.353422 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:15.361268 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-984dc16c-c244-4c45-95f5-d451f7b84ee5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ecc6d950ec1b483d95cfd1af1e34cd39'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:15.362586 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-984dc16c-c244-4c45-95f5-d451f7b84ee5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ecc6d950ec1b483d95cfd1af1e34cd39'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:15.369863 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1080/2130] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:13:15 2022] DELETE /identity/v3/users/ecc6d950ec1b483d95cfd1af1e34cd39 => generated 0 bytes in 137 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:13:15.374644 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-926dcf6b-8d3b-4c83-bacf-74626c147fde None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:15.375202 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-926dcf6b-8d3b-4c83-bacf-74626c147fde None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:15.375418 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-926dcf6b-8d3b-4c83-bacf-74626c147fde None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:15.395735 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-926dcf6b-8d3b-4c83-bacf-74626c147fde None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:13:15.397122 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-926dcf6b-8d3b-4c83-bacf-74626c147fde None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:15.493921 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1051/2131] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:13:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:13:15.504644 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-984dc16c-c244-4c45-95f5-d451f7b84ee5 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:15.506151 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-984dc16c-c244-4c45-95f5-d451f7b84ee5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:15.567708 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-984dc16c-c244-4c45-95f5-d451f7b84ee5 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:15.571852 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9916feaa-59c0-456b-b6ad-5233206a2fdd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:15.572660 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9916feaa-59c0-456b-b6ad-5233206a2fdd service neutron] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:15.572941 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9916feaa-59c0-456b-b6ad-5233206a2fdd service neutron] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:15.573210 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9916feaa-59c0-456b-b6ad-5233206a2fdd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:15.588876 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9916feaa-59c0-456b-b6ad-5233206a2fdd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:15.590326 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9916feaa-59c0-456b-b6ad-5233206a2fdd service neutron] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:15.613907 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1081/2132] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:13:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:13:15.912588 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-926dcf6b-8d3b-4c83-bacf-74626c147fde None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:15.914861 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-926dcf6b-8d3b-4c83-bacf-74626c147fde None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:15.930095 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-926dcf6b-8d3b-4c83-bacf-74626c147fde None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:15.939601 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-130f8695-3843-4571-8a2a-0c74e86fa529 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:15.940740 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-130f8695-3843-4571-8a2a-0c74e86fa529 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:15.941134 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-130f8695-3843-4571-8a2a-0c74e86fa529 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:15.941664 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-130f8695-3843-4571-8a2a-0c74e86fa529 admin admin] PATH_INFO: `/v3/projects/d290987cdcab450a8ee193099b816fd3` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:15.947581 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-130f8695-3843-4571-8a2a-0c74e86fa529 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d290987cdcab450a8ee193099b816fd3)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:15.953017 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-130f8695-3843-4571-8a2a-0c74e86fa529 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:16.006149 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-130f8695-3843-4571-8a2a-0c74e86fa529 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd290987cdcab450a8ee193099b816fd3', 'request_id': 'req-130f8695-3843-4571-8a2a-0c74e86fa529'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:16.015927 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-130f8695-3843-4571-8a2a-0c74e86fa529 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd290987cdcab450a8ee193099b816fd3', 'request_id': 'req-130f8695-3843-4571-8a2a-0c74e86fa529'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:16.023268 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-130f8695-3843-4571-8a2a-0c74e86fa529 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd290987cdcab450a8ee193099b816fd3', 'request_id': 'req-130f8695-3843-4571-8a2a-0c74e86fa529'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:16.036298 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-130f8695-3843-4571-8a2a-0c74e86fa529 admin admin] The token cache is being invalidate because project d290987cdcab450a8ee193099b816fd3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81624) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:13:16.036663 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-130f8695-3843-4571-8a2a-0c74e86fa529 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:16.037433 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-130f8695-3843-4571-8a2a-0c74e86fa529 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:16.039386 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1052/2133] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:13:15 2022] DELETE /identity/v3/projects/d290987cdcab450a8ee193099b816fd3 => generated 0 bytes in 127 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:13:16.045294 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9916feaa-59c0-456b-b6ad-5233206a2fdd service neutron] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:16.057111 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9916feaa-59c0-456b-b6ad-5233206a2fdd service neutron] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:16.123558 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-075e7754-dc1e-4a63-9574-25c0da5547a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:16.124261 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-075e7754-dc1e-4a63-9574-25c0da5547a2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:16.124574 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-075e7754-dc1e-4a63-9574-25c0da5547a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:16.124847 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-075e7754-dc1e-4a63-9574-25c0da5547a2 admin admin] PATH_INFO: `/v3/users/f8ffc73545db4ee4af3cddc8bfa1b781` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:16.126148 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-075e7754-dc1e-4a63-9574-25c0da5547a2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f8ffc73545db4ee4af3cddc8bfa1b781)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:16.128196 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-075e7754-dc1e-4a63-9574-25c0da5547a2 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:16.204216 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-075e7754-dc1e-4a63-9574-25c0da5547a2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f8ffc73545db4ee4af3cddc8bfa1b781'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:16.218166 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-075e7754-dc1e-4a63-9574-25c0da5547a2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f8ffc73545db4ee4af3cddc8bfa1b781'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:16.247163 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-075e7754-dc1e-4a63-9574-25c0da5547a2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f8ffc73545db4ee4af3cddc8bfa1b781'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:16.264773 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-075e7754-dc1e-4a63-9574-25c0da5547a2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f8ffc73545db4ee4af3cddc8bfa1b781'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:16.265799 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-075e7754-dc1e-4a63-9574-25c0da5547a2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f8ffc73545db4ee4af3cddc8bfa1b781'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:16.293652 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1082/2134] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:13:16 2022] DELETE /identity/v3/users/f8ffc73545db4ee4af3cddc8bfa1b781 => generated 0 bytes in 248 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 05:13:16.843890 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-130f8695-3843-4571-8a2a-0c74e86fa529 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:16.845663 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-130f8695-3843-4571-8a2a-0c74e86fa529 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:16.867770 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-130f8695-3843-4571-8a2a-0c74e86fa529 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:16.951860 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c423254-4dcf-412d-b0f4-049446ce3ccd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:16.952656 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c423254-4dcf-412d-b0f4-049446ce3ccd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:16.952939 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c423254-4dcf-412d-b0f4-049446ce3ccd admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:16.953254 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c423254-4dcf-412d-b0f4-049446ce3ccd admin admin] PATH_INFO: `/v3/projects/0bbdf5ad881a488eb068c16d5bde6646` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:16.954594 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c423254-4dcf-412d-b0f4-049446ce3ccd admin admin] RBAC: Authorizing `identity:delete_project(project_id=0bbdf5ad881a488eb068c16d5bde6646)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:16.956617 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c423254-4dcf-412d-b0f4-049446ce3ccd admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:16.959690 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:17.031928 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-7c423254-4dcf-412d-b0f4-049446ce3ccd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0bbdf5ad881a488eb068c16d5bde6646', 'request_id': 'req-7c423254-4dcf-412d-b0f4-049446ce3ccd'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:17.046077 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-7c423254-4dcf-412d-b0f4-049446ce3ccd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0bbdf5ad881a488eb068c16d5bde6646', 'request_id': 'req-7c423254-4dcf-412d-b0f4-049446ce3ccd'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:17.056093 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-7c423254-4dcf-412d-b0f4-049446ce3ccd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0bbdf5ad881a488eb068c16d5bde6646', 'request_id': 'req-7c423254-4dcf-412d-b0f4-049446ce3ccd'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:17.070716 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-7c423254-4dcf-412d-b0f4-049446ce3ccd admin admin] The token cache is being invalidate because project 0bbdf5ad881a488eb068c16d5bde6646 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81624) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:13:17.071045 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-7c423254-4dcf-412d-b0f4-049446ce3ccd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:17.071895 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-7c423254-4dcf-412d-b0f4-049446ce3ccd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:17.073472 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1053/2135] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:13:16 2022] DELETE /identity/v3/projects/0bbdf5ad881a488eb068c16d5bde6646 => generated 0 bytes in 230 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:13:17.077578 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-075e7754-dc1e-4a63-9574-25c0da5547a2 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:17.079059 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-075e7754-dc1e-4a63-9574-25c0da5547a2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:17.093314 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-075e7754-dc1e-4a63-9574-25c0da5547a2 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:17.104633 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:17.194027 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b2611b0-5b2f-43d5-8835-e873618048d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:17.194834 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b2611b0-5b2f-43d5-8835-e873618048d9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:17.195254 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b2611b0-5b2f-43d5-8835-e873618048d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:17.195634 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b2611b0-5b2f-43d5-8835-e873618048d9 admin admin] PATH_INFO: `/v3/users/80d4100aee7a4b86af5ae4f2baae4679` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:17.197208 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b2611b0-5b2f-43d5-8835-e873618048d9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=80d4100aee7a4b86af5ae4f2baae4679)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:17.199323 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b2611b0-5b2f-43d5-8835-e873618048d9 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:17.281917 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-2b2611b0-5b2f-43d5-8835-e873618048d9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '80d4100aee7a4b86af5ae4f2baae4679'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:17.292560 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-2b2611b0-5b2f-43d5-8835-e873618048d9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '80d4100aee7a4b86af5ae4f2baae4679'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:17.311050 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-2b2611b0-5b2f-43d5-8835-e873618048d9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '80d4100aee7a4b86af5ae4f2baae4679'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:17.319579 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-2b2611b0-5b2f-43d5-8835-e873618048d9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '80d4100aee7a4b86af5ae4f2baae4679'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:17.320394 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-2b2611b0-5b2f-43d5-8835-e873618048d9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '80d4100aee7a4b86af5ae4f2baae4679'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:17.329078 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1083/2136] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:13:17 2022] DELETE /identity/v3/users/80d4100aee7a4b86af5ae4f2baae4679 => generated 0 bytes in 252 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:13:17.439734 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c423254-4dcf-412d-b0f4-049446ce3ccd admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:17.441957 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-7c423254-4dcf-412d-b0f4-049446ce3ccd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:17.473100 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c423254-4dcf-412d-b0f4-049446ce3ccd admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:17.550877 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a05794e-8c0a-452a-8190-587bbbbb8ffd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:17.551528 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a05794e-8c0a-452a-8190-587bbbbb8ffd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:17.551810 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a05794e-8c0a-452a-8190-587bbbbb8ffd admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:17.552084 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a05794e-8c0a-452a-8190-587bbbbb8ffd admin admin] PATH_INFO: `/v3/users/d611a49b03554b448a09fe86604af02b` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:17.553870 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a05794e-8c0a-452a-8190-587bbbbb8ffd admin admin] RBAC: Authorizing `identity:delete_user(user_id=d611a49b03554b448a09fe86604af02b)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:17.556061 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a05794e-8c0a-452a-8190-587bbbbb8ffd admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:17.635072 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-4a05794e-8c0a-452a-8190-587bbbbb8ffd admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd611a49b03554b448a09fe86604af02b'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:17.636510 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-4a05794e-8c0a-452a-8190-587bbbbb8ffd admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd611a49b03554b448a09fe86604af02b'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:17.637429 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-4a05794e-8c0a-452a-8190-587bbbbb8ffd admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd611a49b03554b448a09fe86604af02b'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:17.653561 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-4a05794e-8c0a-452a-8190-587bbbbb8ffd admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd611a49b03554b448a09fe86604af02b'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:17.661621 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-4a05794e-8c0a-452a-8190-587bbbbb8ffd admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd611a49b03554b448a09fe86604af02b'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:17.675309 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1054/2137] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:13:17 2022] DELETE /identity/v3/users/d611a49b03554b448a09fe86604af02b => generated 0 bytes in 236 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:13:17.727540 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b2611b0-5b2f-43d5-8835-e873618048d9 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:17.728953 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-2b2611b0-5b2f-43d5-8835-e873618048d9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:17.754224 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b2611b0-5b2f-43d5-8835-e873618048d9 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:17.848444 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57fadc92-ed3a-4483-a3a5-1e31a5d650eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:17.849092 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57fadc92-ed3a-4483-a3a5-1e31a5d650eb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:17.849369 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57fadc92-ed3a-4483-a3a5-1e31a5d650eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:17.849639 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57fadc92-ed3a-4483-a3a5-1e31a5d650eb admin admin] PATH_INFO: `/v3/projects/bd0607128d5f4ece9192e93377e6f729` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:17.851143 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57fadc92-ed3a-4483-a3a5-1e31a5d650eb admin admin] RBAC: Authorizing `identity:delete_project(project_id=bd0607128d5f4ece9192e93377e6f729)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:17.853057 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57fadc92-ed3a-4483-a3a5-1e31a5d650eb admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:17.892901 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:17.903800 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a05794e-8c0a-452a-8190-587bbbbb8ffd admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:17.907021 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-4a05794e-8c0a-452a-8190-587bbbbb8ffd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:17.913422 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-57fadc92-ed3a-4483-a3a5-1e31a5d650eb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bd0607128d5f4ece9192e93377e6f729', 'request_id': 'req-57fadc92-ed3a-4483-a3a5-1e31a5d650eb'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:17.922323 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-57fadc92-ed3a-4483-a3a5-1e31a5d650eb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bd0607128d5f4ece9192e93377e6f729', 'request_id': 'req-57fadc92-ed3a-4483-a3a5-1e31a5d650eb'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:17.930854 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a05794e-8c0a-452a-8190-587bbbbb8ffd admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:17.940814 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:17.946887 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-57fadc92-ed3a-4483-a3a5-1e31a5d650eb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bd0607128d5f4ece9192e93377e6f729', 'request_id': 'req-57fadc92-ed3a-4483-a3a5-1e31a5d650eb'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:17.960052 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-57fadc92-ed3a-4483-a3a5-1e31a5d650eb admin admin] The token cache is being invalidate because project bd0607128d5f4ece9192e93377e6f729 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81625) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:13:17.960348 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-57fadc92-ed3a-4483-a3a5-1e31a5d650eb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:17.961186 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-57fadc92-ed3a-4483-a3a5-1e31a5d650eb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:17.962985 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1084/2138] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:13:17 2022] DELETE /identity/v3/projects/bd0607128d5f4ece9192e93377e6f729 => generated 0 bytes in 236 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:13:17.969866 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b800ef-6041-4830-b6be-42cc468e2b5a None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:17.970096 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b800ef-6041-4830-b6be-42cc468e2b5a None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:17.970671 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b800ef-6041-4830-b6be-42cc468e2b5a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:17.997672 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-d8b800ef-6041-4830-b6be-42cc468e2b5a None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:13:17.999239 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-d8b800ef-6041-4830-b6be-42cc468e2b5a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:18.002343 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cafb222-4e3a-45bf-af02-051c0ede203c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:18.003203 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cafb222-4e3a-45bf-af02-051c0ede203c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:18.003766 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cafb222-4e3a-45bf-af02-051c0ede203c admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:18.004180 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cafb222-4e3a-45bf-af02-051c0ede203c admin admin] PATH_INFO: `/v3/projects/0ed43db2f368485990e3add370edae5f` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:18.005424 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cafb222-4e3a-45bf-af02-051c0ede203c admin admin] RBAC: Authorizing `identity:delete_project(project_id=0ed43db2f368485990e3add370edae5f)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:18.007528 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cafb222-4e3a-45bf-af02-051c0ede203c admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:18.008062 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1085/2139] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:13:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:13:18.017387 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31bb7ae5-fd46-4594-8aeb-a5b4f2a87341 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:18.017897 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31bb7ae5-fd46-4594-8aeb-a5b4f2a87341 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:18.018246 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31bb7ae5-fd46-4594-8aeb-a5b4f2a87341 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:18.044600 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-31bb7ae5-fd46-4594-8aeb-a5b4f2a87341 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:13:18.047291 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-31bb7ae5-fd46-4594-8aeb-a5b4f2a87341 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:18.062037 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1086/2140] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:13:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:13:18.063438 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31bb7ae5-fd46-4594-8aeb-a5b4f2a87341 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:18.071936 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-2cafb222-4e3a-45bf-af02-051c0ede203c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0ed43db2f368485990e3add370edae5f', 'request_id': 'req-2cafb222-4e3a-45bf-af02-051c0ede203c'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:18.072756 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31bb7ae5-fd46-4594-8aeb-a5b4f2a87341 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:18.081119 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-2cafb222-4e3a-45bf-af02-051c0ede203c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0ed43db2f368485990e3add370edae5f', 'request_id': 'req-2cafb222-4e3a-45bf-af02-051c0ede203c'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:18.107416 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-2cafb222-4e3a-45bf-af02-051c0ede203c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0ed43db2f368485990e3add370edae5f', 'request_id': 'req-2cafb222-4e3a-45bf-af02-051c0ede203c'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:18.128313 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-2cafb222-4e3a-45bf-af02-051c0ede203c admin admin] The token cache is being invalidate because project 0ed43db2f368485990e3add370edae5f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81624) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:13:18.128882 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-2cafb222-4e3a-45bf-af02-051c0ede203c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:18.130002 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-2cafb222-4e3a-45bf-af02-051c0ede203c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:18.134334 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1055/2141] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:13:17 2022] DELETE /identity/v3/projects/0ed43db2f368485990e3add370edae5f => generated 0 bytes in 231 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 05:13:18.138998 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90172a92-2378-4a64-8804-3c76961d02fc None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:18.139670 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90172a92-2378-4a64-8804-3c76961d02fc None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:18.140091 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90172a92-2378-4a64-8804-3c76961d02fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:18.141239 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92357341-e672-4980-be53-1e2afa54f201 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:18.141964 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92357341-e672-4980-be53-1e2afa54f201 admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:18.142365 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92357341-e672-4980-be53-1e2afa54f201 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:18.142745 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92357341-e672-4980-be53-1e2afa54f201 admin admin] PATH_INFO: `/v3/domains` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:18.144043 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92357341-e672-4980-be53-1e2afa54f201 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:13:18.144493 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92357341-e672-4980-be53-1e2afa54f201 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:18.146212 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92357341-e672-4980-be53-1e2afa54f201 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:18.167907 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1087/2142] 10.222.0.77 () {58 vars in 1153 bytes} [Tue Aug 30 05:13:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 105 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:13:18.169153 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92357341-e672-4980-be53-1e2afa54f201 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:18.171421 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-92357341-e672-4980-be53-1e2afa54f201 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:18.176036 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-90172a92-2378-4a64-8804-3c76961d02fc None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:13:18.177754 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-90172a92-2378-4a64-8804-3c76961d02fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:18.188116 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92357341-e672-4980-be53-1e2afa54f201 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:18.188312 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1056/2143] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:13:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 50 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:13:18.196067 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35c4dc89-37bc-46ae-970e-bc43115db818 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:18.196472 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35c4dc89-37bc-46ae-970e-bc43115db818 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:18.197298 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35c4dc89-37bc-46ae-970e-bc43115db818 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:18.198029 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1f840d7-0462-44a3-8707-9fb21bfb427d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:18.198985 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1f840d7-0462-44a3-8707-9fb21bfb427d admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:18.199276 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1f840d7-0462-44a3-8707-9fb21bfb427d admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:18.199656 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1f840d7-0462-44a3-8707-9fb21bfb427d admin admin] PATH_INFO: `/v3/projects` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:18.200171 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1f840d7-0462-44a3-8707-9fb21bfb427d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:18.202087 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1f840d7-0462-44a3-8707-9fb21bfb427d admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:18.223870 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-35c4dc89-37bc-46ae-970e-bc43115db818 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:13:18.225547 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-35c4dc89-37bc-46ae-970e-bc43115db818 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:18.237924 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1057/2144] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:13:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 05:13:18.239445 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35c4dc89-37bc-46ae-970e-bc43115db818 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:18.253994 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35c4dc89-37bc-46ae-970e-bc43115db818 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:18.258854 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1088/2145] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:13:18 2022] POST /identity/v3/projects => generated 365 bytes in 88 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:13:18.262152 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1f840d7-0462-44a3-8707-9fb21bfb427d admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:18.264457 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4280498d-fbed-4383-b64a-2318bef65374 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:18.265065 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4280498d-fbed-4383-b64a-2318bef65374 admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:18.265337 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4280498d-fbed-4383-b64a-2318bef65374 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:18.265604 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4280498d-fbed-4383-b64a-2318bef65374 admin admin] PATH_INFO: `/v3/domains` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:18.266253 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4280498d-fbed-4383-b64a-2318bef65374 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:13:18.266547 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4280498d-fbed-4383-b64a-2318bef65374 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:18.269106 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4280498d-fbed-4383-b64a-2318bef65374 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:18.284648 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1058/2146] 10.222.0.77 () {58 vars in 1153 bytes} [Tue Aug 30 05:13:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:13:18.289069 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4280498d-fbed-4383-b64a-2318bef65374 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:18.303483 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4280498d-fbed-4383-b64a-2318bef65374 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:18.341684 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1f840d7-0462-44a3-8707-9fb21bfb427d admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:18.347905 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bfa671a-e3f1-4e70-b600-b91fbe32e59d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:18.348509 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bfa671a-e3f1-4e70-b600-b91fbe32e59d admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:18.348787 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bfa671a-e3f1-4e70-b600-b91fbe32e59d admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:18.349058 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bfa671a-e3f1-4e70-b600-b91fbe32e59d admin admin] PATH_INFO: `/v3/users` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:18.349683 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bfa671a-e3f1-4e70-b600-b91fbe32e59d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:18.351784 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bfa671a-e3f1-4e70-b600-b91fbe32e59d admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:18.379458 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1089/2147] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:13:18 2022] POST /identity/v3/users => generated 283 bytes in 119 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 05:13:18.383551 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bfa671a-e3f1-4e70-b600-b91fbe32e59d admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:18.387774 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a77cbef7-0000-4741-8487-7ad071262463 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:18.388074 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a77cbef7-0000-4741-8487-7ad071262463 admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:18.388447 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a77cbef7-0000-4741-8487-7ad071262463 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:18.389076 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a77cbef7-0000-4741-8487-7ad071262463 admin admin] PATH_INFO: `/v3/projects` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:18.390625 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a77cbef7-0000-4741-8487-7ad071262463 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:18.394316 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a77cbef7-0000-4741-8487-7ad071262463 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:18.394716 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bfa671a-e3f1-4e70-b600-b91fbe32e59d admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:18.400045 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f40ff780-a63a-44e5-acc1-332426e9a764 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:18.400670 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f40ff780-a63a-44e5-acc1-332426e9a764 admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:18.400989 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f40ff780-a63a-44e5-acc1-332426e9a764 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:18.401249 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f40ff780-a63a-44e5-acc1-332426e9a764 admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:18.401889 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f40ff780-a63a-44e5-acc1-332426e9a764 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:18.403535 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f40ff780-a63a-44e5-acc1-332426e9a764 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:18.406336 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=90,keystone:DELETE=5,keystone:INSERT=5 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:18.418330 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1090/2148] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:13:18 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:13:18.421369 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:18.423217 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f40ff780-a63a-44e5-acc1-332426e9a764 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:18.430916 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1059/2149] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:13:18 2022] POST /identity/v3/projects => generated 365 bytes in 141 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 05:13:18.434348 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a77cbef7-0000-4741-8487-7ad071262463 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:18.442095 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f40ff780-a63a-44e5-acc1-332426e9a764 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:18.510610 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a77cbef7-0000-4741-8487-7ad071262463 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:18.518665 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d35e74ef-663d-481c-8271-4b0c5710a4dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:18.519722 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d35e74ef-663d-481c-8271-4b0c5710a4dd admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:18.521487 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eb52b99-00d7-4168-b14d-de3583f4d9e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:18.521631 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb52b99-00d7-4168-b14d-de3583f4d9e1 admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:18.522037 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d35e74ef-663d-481c-8271-4b0c5710a4dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:18.522216 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb52b99-00d7-4168-b14d-de3583f4d9e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:18.522884 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb52b99-00d7-4168-b14d-de3583f4d9e1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:18.523156 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eb52b99-00d7-4168-b14d-de3583f4d9e1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:18.524909 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eb52b99-00d7-4168-b14d-de3583f4d9e1 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:18.533871 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d35e74ef-663d-481c-8271-4b0c5710a4dd admin admin] PATH_INFO: `/v3/users` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:18.535217 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d35e74ef-663d-481c-8271-4b0c5710a4dd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:18.537513 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d35e74ef-663d-481c-8271-4b0c5710a4dd admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:18.540186 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1091/2150] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:13:18 2022] GET /identity/v3/roles => generated 1420 bytes in 117 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:13:18.541291 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eb52b99-00d7-4168-b14d-de3583f4d9e1 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:18.554153 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eb52b99-00d7-4168-b14d-de3583f4d9e1 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:18.560461 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ba556bf-fade-48ff-b5ea-1d99a5be0104 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:18.561515 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba556bf-fade-48ff-b5ea-1d99a5be0104 admin admin] REQUEST_METHOD: `PUT` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:18.561515 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba556bf-fade-48ff-b5ea-1d99a5be0104 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:18.561692 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba556bf-fade-48ff-b5ea-1d99a5be0104 admin admin] PATH_INFO: `/v3/projects/e2b36feed0b64ff0aec64cb4a66e314a/users/cda7cc6f157744d097d3fce4f5b3d5e5/roles/6f03d86980c34c36b44a8d6764ff4fda` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:18.568133 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1060/2151] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:13:18 2022] POST /identity/v3/users => generated 283 bytes in 135 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:13:18.571921 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d35e74ef-663d-481c-8271-4b0c5710a4dd admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:18.583428 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d35e74ef-663d-481c-8271-4b0c5710a4dd admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:18.583920 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ba556bf-fade-48ff-b5ea-1d99a5be0104 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e2b36feed0b64ff0aec64cb4a66e314a, user_id=cda7cc6f157744d097d3fce4f5b3d5e5, role_id=6f03d86980c34c36b44a8d6764ff4fda)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:18.587228 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ba556bf-fade-48ff-b5ea-1d99a5be0104 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:18.589874 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41df1f4e-a8a9-4579-af65-bf4d46ecab73 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:18.590658 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41df1f4e-a8a9-4579-af65-bf4d46ecab73 admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:18.591026 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41df1f4e-a8a9-4579-af65-bf4d46ecab73 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:18.591388 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41df1f4e-a8a9-4579-af65-bf4d46ecab73 admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:18.592163 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41df1f4e-a8a9-4579-af65-bf4d46ecab73 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:18.594182 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41df1f4e-a8a9-4579-af65-bf4d46ecab73 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:18.610871 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1061/2152] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:13:18 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 05:13:18.613817 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41df1f4e-a8a9-4579-af65-bf4d46ecab73 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:18.617046 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1092/2153] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:13:18 2022] PUT /identity/v3/projects/e2b36feed0b64ff0aec64cb4a66e314a/users/cda7cc6f157744d097d3fce4f5b3d5e5/roles/6f03d86980c34c36b44a8d6764ff4fda => generated 0 bytes in 76 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:13:18.624564 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41df1f4e-a8a9-4579-af65-bf4d46ecab73 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:18.628852 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0adbde9-a029-476b-8fd2-c2c37e759729 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:18.629153 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0adbde9-a029-476b-8fd2-c2c37e759729 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:18.629414 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0adbde9-a029-476b-8fd2-c2c37e759729 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:18.660256 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-f0adbde9-a029-476b-8fd2-c2c37e759729 None None] MFA Rules not processed for user `cda7cc6f157744d097d3fce4f5b3d5e5`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:13:18.661488 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-f0adbde9-a029-476b-8fd2-c2c37e759729 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:18.708029 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5bb93ab-ad03-442e-b32f-ba1d5cf07443 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:18.708648 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5bb93ab-ad03-442e-b32f-ba1d5cf07443 admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:18.708925 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5bb93ab-ad03-442e-b32f-ba1d5cf07443 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:18.709270 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5bb93ab-ad03-442e-b32f-ba1d5cf07443 admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:18.709814 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5bb93ab-ad03-442e-b32f-ba1d5cf07443 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:18.712024 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5bb93ab-ad03-442e-b32f-ba1d5cf07443 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:18.737456 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1062/2154] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:13:18 2022] GET /identity/v3/roles => generated 1420 bytes in 122 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:13:18.740106 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5bb93ab-ad03-442e-b32f-ba1d5cf07443 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:18.749607 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1093/2155] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:13:18 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 05:13:18.750651 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5bb93ab-ad03-442e-b32f-ba1d5cf07443 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:18.754124 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0adbde9-a029-476b-8fd2-c2c37e759729 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:18.756023 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-027c3641-fbaa-4aa1-adf5-5af305df0391 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:18.756659 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-027c3641-fbaa-4aa1-adf5-5af305df0391 admin admin] REQUEST_METHOD: `PUT` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:18.756940 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-027c3641-fbaa-4aa1-adf5-5af305df0391 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:18.757212 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-027c3641-fbaa-4aa1-adf5-5af305df0391 admin admin] PATH_INFO: `/v3/projects/e656eb9012c441398c8ad5bdcc6c4046/users/f4322a87e0d441608afa20c69992eb3a/roles/6f03d86980c34c36b44a8d6764ff4fda` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:18.764774 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0adbde9-a029-476b-8fd2-c2c37e759729 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:18.770505 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51719189-4298-4a74-a8ee-3ecc1a689f70 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:18.771140 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51719189-4298-4a74-a8ee-3ecc1a689f70 admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:18.771437 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51719189-4298-4a74-a8ee-3ecc1a689f70 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:18.771684 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51719189-4298-4a74-a8ee-3ecc1a689f70 admin admin] PATH_INFO: `/v3/projects` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:18.772309 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51719189-4298-4a74-a8ee-3ecc1a689f70 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:18.774248 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51719189-4298-4a74-a8ee-3ecc1a689f70 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:18.780609 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-027c3641-fbaa-4aa1-adf5-5af305df0391 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e656eb9012c441398c8ad5bdcc6c4046, user_id=f4322a87e0d441608afa20c69992eb3a, role_id=6f03d86980c34c36b44a8d6764ff4fda)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:18.783949 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-027c3641-fbaa-4aa1-adf5-5af305df0391 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:18.809037 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1063/2156] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:13:18 2022] PUT /identity/v3/projects/e656eb9012c441398c8ad5bdcc6c4046/users/f4322a87e0d441608afa20c69992eb3a/roles/6f03d86980c34c36b44a8d6764ff4fda => generated 0 bytes in 69 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:13:18.816229 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1094/2157] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:13:18 2022] POST /identity/v3/projects => generated 367 bytes in 59 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 05:13:18.817855 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-027c3641-fbaa-4aa1-adf5-5af305df0391 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:18.822722 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eaf9730-b875-4ab8-b902-b52a098cae18 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:18.824953 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eaf9730-b875-4ab8-b902-b52a098cae18 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:18.825175 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eaf9730-b875-4ab8-b902-b52a098cae18 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:18.854403 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-3eaf9730-b875-4ab8-b902-b52a098cae18 None None] MFA Rules not processed for user `f4322a87e0d441608afa20c69992eb3a`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:13:18.855658 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-3eaf9730-b875-4ab8-b902-b52a098cae18 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:18.891480 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:18.897549 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:18.919639 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-027c3641-fbaa-4aa1-adf5-5af305df0391 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:18.944148 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1095/2158] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:13:18 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 05:13:18.955501 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eaf9730-b875-4ab8-b902-b52a098cae18 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:18.957160 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-3eaf9730-b875-4ab8-b902-b52a098cae18 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:19.010015 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84086fe9-0a0b-47cf-ad4f-072fba686153 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:19.010878 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84086fe9-0a0b-47cf-ad4f-072fba686153 admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:19.011301 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84086fe9-0a0b-47cf-ad4f-072fba686153 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:19.011743 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84086fe9-0a0b-47cf-ad4f-072fba686153 admin admin] PATH_INFO: `/v3/users` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:19.013191 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84086fe9-0a0b-47cf-ad4f-072fba686153 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:19.015584 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84086fe9-0a0b-47cf-ad4f-072fba686153 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:19.035544 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eaf9730-b875-4ab8-b902-b52a098cae18 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:19.039893 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aa8923d-7438-46c9-a42b-3ed8593ce84d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '019a79b2affe478b8a30d1c538d0415c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:19.040697 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aa8923d-7438-46c9-a42b-3ed8593ce84d service cinder] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:19.041075 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aa8923d-7438-46c9-a42b-3ed8593ce84d service cinder] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:19.041460 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aa8923d-7438-46c9-a42b-3ed8593ce84d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:19.045706 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1064/2159] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:13:18 2022] POST /identity/v3/users => generated 290 bytes in 225 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 05:13:19.047611 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84086fe9-0a0b-47cf-ad4f-072fba686153 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:19.058890 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aa8923d-7438-46c9-a42b-3ed8593ce84d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:19.060443 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aa8923d-7438-46c9-a42b-3ed8593ce84d service cinder] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:19.071449 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84086fe9-0a0b-47cf-ad4f-072fba686153 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:19.074148 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1096/2160] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:13:18 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 119 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 05:13:19.079970 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88bd9c3c-88a2-46be-9f5c-eb0df3a10937 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:19.079970 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88bd9c3c-88a2-46be-9f5c-eb0df3a10937 admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:19.079970 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88bd9c3c-88a2-46be-9f5c-eb0df3a10937 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:19.080431 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88bd9c3c-88a2-46be-9f5c-eb0df3a10937 admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:19.080896 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88bd9c3c-88a2-46be-9f5c-eb0df3a10937 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:19.082482 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88bd9c3c-88a2-46be-9f5c-eb0df3a10937 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:19.098536 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1065/2161] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:13:19 2022] GET /identity/v3/roles => generated 1420 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 05:13:19.103391 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aa8923d-7438-46c9-a42b-3ed8593ce84d service cinder] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:19.116998 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aa8923d-7438-46c9-a42b-3ed8593ce84d service cinder] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:19.123332 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29fa7031-e373-446b-af85-9e674eb19a28 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:19.123995 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29fa7031-e373-446b-af85-9e674eb19a28 admin admin] REQUEST_METHOD: `PUT` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:19.124274 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29fa7031-e373-446b-af85-9e674eb19a28 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:19.124550 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29fa7031-e373-446b-af85-9e674eb19a28 admin admin] PATH_INFO: `/v3/projects/2cad4b83d49044e7906e77aa34352b8c/users/e34026af9f7a45d9b699d5be4548d7d1/roles/9d9d5d2c270146488249b7ed943280db` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:19.146276 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29fa7031-e373-446b-af85-9e674eb19a28 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2cad4b83d49044e7906e77aa34352b8c, user_id=e34026af9f7a45d9b699d5be4548d7d1, role_id=9d9d5d2c270146488249b7ed943280db)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:19.151946 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29fa7031-e373-446b-af85-9e674eb19a28 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:19.174695 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1097/2162] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:13:19 2022] PUT /identity/v3/projects/2cad4b83d49044e7906e77aa34352b8c/users/e34026af9f7a45d9b699d5be4548d7d1/roles/9d9d5d2c270146488249b7ed943280db => generated 0 bytes in 72 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:13:19.178908 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88bd9c3c-88a2-46be-9f5c-eb0df3a10937 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:19.197714 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88bd9c3c-88a2-46be-9f5c-eb0df3a10937 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:19.273657 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31b8fc86-8e3a-4146-b0c2-27d3805902c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:19.274500 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31b8fc86-8e3a-4146-b0c2-27d3805902c8 admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:19.274996 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31b8fc86-8e3a-4146-b0c2-27d3805902c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:19.275241 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31b8fc86-8e3a-4146-b0c2-27d3805902c8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:19.275877 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31b8fc86-8e3a-4146-b0c2-27d3805902c8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:19.277500 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31b8fc86-8e3a-4146-b0c2-27d3805902c8 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:19.292314 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1066/2163] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:13:19 2022] GET /identity/v3/roles => generated 1420 bytes in 114 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:13:19.293223 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29fa7031-e373-446b-af85-9e674eb19a28 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:19.305189 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29fa7031-e373-446b-af85-9e674eb19a28 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:19.311402 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8db22b79-ea5b-4f80-a3af-ac27d5e2d732 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:19.311648 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8db22b79-ea5b-4f80-a3af-ac27d5e2d732 admin admin] REQUEST_METHOD: `PUT` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:19.311931 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8db22b79-ea5b-4f80-a3af-ac27d5e2d732 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:19.312203 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8db22b79-ea5b-4f80-a3af-ac27d5e2d732 admin admin] PATH_INFO: `/v3/projects/2cad4b83d49044e7906e77aa34352b8c/users/e34026af9f7a45d9b699d5be4548d7d1/roles/9d9d5d2c270146488249b7ed943280db` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:19.314693 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8db22b79-ea5b-4f80-a3af-ac27d5e2d732 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2cad4b83d49044e7906e77aa34352b8c, user_id=e34026af9f7a45d9b699d5be4548d7d1, role_id=9d9d5d2c270146488249b7ed943280db)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:19.317869 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8db22b79-ea5b-4f80-a3af-ac27d5e2d732 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:19.334873 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1098/2164] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:13:19 2022] PUT /identity/v3/projects/2cad4b83d49044e7906e77aa34352b8c/users/e34026af9f7a45d9b699d5be4548d7d1/roles/9d9d5d2c270146488249b7ed943280db => generated 0 bytes in 42 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:13:19.345553 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1536055-5431-43ce-9796-bf4a6505520b None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:19.345855 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1536055-5431-43ce-9796-bf4a6505520b None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:19.346631 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1536055-5431-43ce-9796-bf4a6505520b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:19.385755 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-d1536055-5431-43ce-9796-bf4a6505520b None None] MFA Rules not processed for user `e34026af9f7a45d9b699d5be4548d7d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:13:19.388203 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-d1536055-5431-43ce-9796-bf4a6505520b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:19.511039 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:19.518008 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1067/2165] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:13:19 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:13:19.530018 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8db22b79-ea5b-4f80-a3af-ac27d5e2d732 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:19.597003 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8db22b79-ea5b-4f80-a3af-ac27d5e2d732 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:19.604298 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d19daa08-2657-474a-9cc0-75886667b6ea service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '019a79b2affe478b8a30d1c538d0415c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:19.604897 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d19daa08-2657-474a-9cc0-75886667b6ea service cinder] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:19.605206 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d19daa08-2657-474a-9cc0-75886667b6ea service cinder] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:19.605864 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d19daa08-2657-474a-9cc0-75886667b6ea service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:19.616456 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d19daa08-2657-474a-9cc0-75886667b6ea service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:19.618040 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d19daa08-2657-474a-9cc0-75886667b6ea service cinder] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:19.628527 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1099/2166] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:13:19 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:13:19.717377 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1536055-5431-43ce-9796-bf4a6505520b None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:19.727905 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1536055-5431-43ce-9796-bf4a6505520b None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:19.731648 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b10fabaf-24db-4f74-8836-4db35090bc59 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '019a79b2affe478b8a30d1c538d0415c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:19.732244 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b10fabaf-24db-4f74-8836-4db35090bc59 service cinder] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:19.732517 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b10fabaf-24db-4f74-8836-4db35090bc59 service cinder] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:19.732845 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b10fabaf-24db-4f74-8836-4db35090bc59 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:19.809912 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b10fabaf-24db-4f74-8836-4db35090bc59 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:19.811742 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b10fabaf-24db-4f74-8836-4db35090bc59 service cinder] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:19.822069 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1068/2167] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:13:19 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:13:24.706919 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d19daa08-2657-474a-9cc0-75886667b6ea service cinder] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:24.747963 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:24.772685 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d19daa08-2657-474a-9cc0-75886667b6ea service cinder] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:24.777176 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd36c8ca-0ba5-483f-b6b6-0515940f7b10 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:24.777795 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd36c8ca-0ba5-483f-b6b6-0515940f7b10 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:24.778074 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd36c8ca-0ba5-483f-b6b6-0515940f7b10 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:24.778344 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd36c8ca-0ba5-483f-b6b6-0515940f7b10 admin admin] PATH_INFO: `/v3/users/cda7cc6f157744d097d3fce4f5b3d5e5` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:24.779479 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd36c8ca-0ba5-483f-b6b6-0515940f7b10 admin admin] RBAC: Authorizing `identity:delete_user(user_id=cda7cc6f157744d097d3fce4f5b3d5e5)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:24.781405 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd36c8ca-0ba5-483f-b6b6-0515940f7b10 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:24.828648 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-fd36c8ca-0ba5-483f-b6b6-0515940f7b10 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cda7cc6f157744d097d3fce4f5b3d5e5'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:24.836337 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-fd36c8ca-0ba5-483f-b6b6-0515940f7b10 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cda7cc6f157744d097d3fce4f5b3d5e5'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:24.848994 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-fd36c8ca-0ba5-483f-b6b6-0515940f7b10 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cda7cc6f157744d097d3fce4f5b3d5e5'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:24.856090 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-fd36c8ca-0ba5-483f-b6b6-0515940f7b10 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cda7cc6f157744d097d3fce4f5b3d5e5'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:24.856722 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-fd36c8ca-0ba5-483f-b6b6-0515940f7b10 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cda7cc6f157744d097d3fce4f5b3d5e5'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:24.863559 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1100/2168] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:13:24 2022] DELETE /identity/v3/users/cda7cc6f157744d097d3fce4f5b3d5e5 => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:13:24.868091 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dade0fd-5c3a-434f-8da0-39e0737cd83b None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:24.868583 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dade0fd-5c3a-434f-8da0-39e0737cd83b None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:24.868687 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dade0fd-5c3a-434f-8da0-39e0737cd83b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:24.887237 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-9dade0fd-5c3a-434f-8da0-39e0737cd83b None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:13:24.888428 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-9dade0fd-5c3a-434f-8da0-39e0737cd83b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:24.948974 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1069/2169] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:13:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:13:24.960665 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd36c8ca-0ba5-483f-b6b6-0515940f7b10 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:24.961752 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-fd36c8ca-0ba5-483f-b6b6-0515940f7b10 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:25.038251 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd36c8ca-0ba5-483f-b6b6-0515940f7b10 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:25.038251 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbec9dec-b7e9-4323-aa4f-bad7b79d954c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:25.038251 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbec9dec-b7e9-4323-aa4f-bad7b79d954c service neutron] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:25.038251 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbec9dec-b7e9-4323-aa4f-bad7b79d954c service neutron] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:25.038695 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbec9dec-b7e9-4323-aa4f-bad7b79d954c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:25.046672 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbec9dec-b7e9-4323-aa4f-bad7b79d954c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:25.049796 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbec9dec-b7e9-4323-aa4f-bad7b79d954c service neutron] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:25.060102 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1101/2170] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:13:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:13:25.354035 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dade0fd-5c3a-434f-8da0-39e0737cd83b None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:25.355878 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-9dade0fd-5c3a-434f-8da0-39e0737cd83b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:25.372337 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dade0fd-5c3a-434f-8da0-39e0737cd83b None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:25.380291 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd83fd1f-a313-45b6-85e0-a1898bc81a12 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:25.381064 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd83fd1f-a313-45b6-85e0-a1898bc81a12 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:25.381446 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd83fd1f-a313-45b6-85e0-a1898bc81a12 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:25.381905 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd83fd1f-a313-45b6-85e0-a1898bc81a12 admin admin] PATH_INFO: `/v3/projects/e2b36feed0b64ff0aec64cb4a66e314a` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:25.383152 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd83fd1f-a313-45b6-85e0-a1898bc81a12 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e2b36feed0b64ff0aec64cb4a66e314a)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:25.386224 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd83fd1f-a313-45b6-85e0-a1898bc81a12 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:25.452590 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-dd83fd1f-a313-45b6-85e0-a1898bc81a12 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e2b36feed0b64ff0aec64cb4a66e314a', 'request_id': 'req-dd83fd1f-a313-45b6-85e0-a1898bc81a12'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:25.465915 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-dd83fd1f-a313-45b6-85e0-a1898bc81a12 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e2b36feed0b64ff0aec64cb4a66e314a', 'request_id': 'req-dd83fd1f-a313-45b6-85e0-a1898bc81a12'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:25.474133 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-dd83fd1f-a313-45b6-85e0-a1898bc81a12 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e2b36feed0b64ff0aec64cb4a66e314a', 'request_id': 'req-dd83fd1f-a313-45b6-85e0-a1898bc81a12'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:25.485706 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-dd83fd1f-a313-45b6-85e0-a1898bc81a12 admin admin] The token cache is being invalidate because project e2b36feed0b64ff0aec64cb4a66e314a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81624) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:13:25.486198 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-dd83fd1f-a313-45b6-85e0-a1898bc81a12 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:25.487759 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-dd83fd1f-a313-45b6-85e0-a1898bc81a12 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:25.489475 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1070/2171] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:13:25 2022] DELETE /identity/v3/projects/e2b36feed0b64ff0aec64cb4a66e314a => generated 0 bytes in 136 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:13:25.493436 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbec9dec-b7e9-4323-aa4f-bad7b79d954c service neutron] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:25.506304 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbec9dec-b7e9-4323-aa4f-bad7b79d954c service neutron] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:25.565950 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:25.574261 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d658f91a-26b7-4f95-a7f5-4409621fc27e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:25.576396 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d658f91a-26b7-4f95-a7f5-4409621fc27e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:25.577349 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d658f91a-26b7-4f95-a7f5-4409621fc27e admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:25.577712 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d658f91a-26b7-4f95-a7f5-4409621fc27e admin admin] PATH_INFO: `/v3/users/e34026af9f7a45d9b699d5be4548d7d1` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:25.579124 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d658f91a-26b7-4f95-a7f5-4409621fc27e admin admin] RBAC: Authorizing `identity:delete_user(user_id=e34026af9f7a45d9b699d5be4548d7d1)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:25.582207 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d658f91a-26b7-4f95-a7f5-4409621fc27e admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:25.638831 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-d658f91a-26b7-4f95-a7f5-4409621fc27e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e34026af9f7a45d9b699d5be4548d7d1'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:25.647361 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-d658f91a-26b7-4f95-a7f5-4409621fc27e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e34026af9f7a45d9b699d5be4548d7d1'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:25.659836 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-d658f91a-26b7-4f95-a7f5-4409621fc27e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e34026af9f7a45d9b699d5be4548d7d1'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:25.666758 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-d658f91a-26b7-4f95-a7f5-4409621fc27e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e34026af9f7a45d9b699d5be4548d7d1'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:25.667409 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-d658f91a-26b7-4f95-a7f5-4409621fc27e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e34026af9f7a45d9b699d5be4548d7d1'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:25.673396 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1102/2172] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:13:25 2022] DELETE /identity/v3/users/e34026af9f7a45d9b699d5be4548d7d1 => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:13:26.018791 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd83fd1f-a313-45b6-85e0-a1898bc81a12 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:26.020419 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-dd83fd1f-a313-45b6-85e0-a1898bc81a12 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:26.034251 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd83fd1f-a313-45b6-85e0-a1898bc81a12 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:26.054184 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:26.107171 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c61283a3-e381-44d9-8fe1-d2a97e43852b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:26.107809 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c61283a3-e381-44d9-8fe1-d2a97e43852b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:26.108093 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c61283a3-e381-44d9-8fe1-d2a97e43852b admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:26.108364 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c61283a3-e381-44d9-8fe1-d2a97e43852b admin admin] PATH_INFO: `/v3/projects/2cad4b83d49044e7906e77aa34352b8c` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:26.109630 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c61283a3-e381-44d9-8fe1-d2a97e43852b admin admin] RBAC: Authorizing `identity:delete_project(project_id=2cad4b83d49044e7906e77aa34352b8c)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:26.111957 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c61283a3-e381-44d9-8fe1-d2a97e43852b admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:26.170635 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-c61283a3-e381-44d9-8fe1-d2a97e43852b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2cad4b83d49044e7906e77aa34352b8c', 'request_id': 'req-c61283a3-e381-44d9-8fe1-d2a97e43852b'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:26.180837 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-c61283a3-e381-44d9-8fe1-d2a97e43852b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2cad4b83d49044e7906e77aa34352b8c', 'request_id': 'req-c61283a3-e381-44d9-8fe1-d2a97e43852b'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:26.194405 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-c61283a3-e381-44d9-8fe1-d2a97e43852b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2cad4b83d49044e7906e77aa34352b8c', 'request_id': 'req-c61283a3-e381-44d9-8fe1-d2a97e43852b'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:26.205075 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-c61283a3-e381-44d9-8fe1-d2a97e43852b admin admin] The token cache is being invalidate because project 2cad4b83d49044e7906e77aa34352b8c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81624) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:13:26.205908 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-c61283a3-e381-44d9-8fe1-d2a97e43852b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:26.206661 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-c61283a3-e381-44d9-8fe1-d2a97e43852b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:26.208306 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1071/2173] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:13:26 2022] DELETE /identity/v3/projects/2cad4b83d49044e7906e77aa34352b8c => generated 0 bytes in 191 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:13:26.214098 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86e5caf6-10d0-419f-83cb-b582efa27014 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:26.214584 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86e5caf6-10d0-419f-83cb-b582efa27014 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:26.214676 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86e5caf6-10d0-419f-83cb-b582efa27014 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:26.235430 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-86e5caf6-10d0-419f-83cb-b582efa27014 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:13:26.236691 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-86e5caf6-10d0-419f-83cb-b582efa27014 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:26.300784 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1103/2174] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:13:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 05:13:26.308552 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da5876b2-868c-4620-acc0-7a0623aafd2e None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:26.308762 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da5876b2-868c-4620-acc0-7a0623aafd2e None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:26.309037 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da5876b2-868c-4620-acc0-7a0623aafd2e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:26.330761 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-da5876b2-868c-4620-acc0-7a0623aafd2e None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:13:26.331997 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-da5876b2-868c-4620-acc0-7a0623aafd2e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:26.339920 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1072/2175] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:13:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:13:26.344414 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86e5caf6-10d0-419f-83cb-b582efa27014 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:26.355092 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86e5caf6-10d0-419f-83cb-b582efa27014 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:26.359636 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e75af9c-3dab-4f8c-ae2a-2210ba75f875 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:26.360226 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e75af9c-3dab-4f8c-ae2a-2210ba75f875 admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:26.360504 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e75af9c-3dab-4f8c-ae2a-2210ba75f875 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:26.360774 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e75af9c-3dab-4f8c-ae2a-2210ba75f875 admin admin] PATH_INFO: `/v3/domains` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:26.361928 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e75af9c-3dab-4f8c-ae2a-2210ba75f875 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:13:26.362236 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e75af9c-3dab-4f8c-ae2a-2210ba75f875 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:26.363941 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e75af9c-3dab-4f8c-ae2a-2210ba75f875 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:26.375857 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1104/2176] 10.222.0.77 () {58 vars in 1153 bytes} [Tue Aug 30 05:13:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:13:26.381332 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da5876b2-868c-4620-acc0-7a0623aafd2e None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:26.390472 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da5876b2-868c-4620-acc0-7a0623aafd2e None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:26.395387 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d657f82-9944-4317-ae48-a38e8a1b5531 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:26.395975 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d657f82-9944-4317-ae48-a38e8a1b5531 admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:26.396255 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d657f82-9944-4317-ae48-a38e8a1b5531 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:26.396532 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d657f82-9944-4317-ae48-a38e8a1b5531 admin admin] PATH_INFO: `/v3/projects` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:26.397150 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d657f82-9944-4317-ae48-a38e8a1b5531 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:26.399072 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d657f82-9944-4317-ae48-a38e8a1b5531 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:26.425762 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1073/2177] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:13:26 2022] POST /identity/v3/projects => generated 383 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:13:26.430054 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e75af9c-3dab-4f8c-ae2a-2210ba75f875 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:26.440775 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e75af9c-3dab-4f8c-ae2a-2210ba75f875 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:26.499560 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35e362c4-c36b-4309-9497-1f090c508ece admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:26.500152 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e362c4-c36b-4309-9497-1f090c508ece admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:26.500427 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e362c4-c36b-4309-9497-1f090c508ece admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:26.500696 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e362c4-c36b-4309-9497-1f090c508ece admin admin] PATH_INFO: `/v3/users` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:26.501320 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35e362c4-c36b-4309-9497-1f090c508ece admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:26.503791 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35e362c4-c36b-4309-9497-1f090c508ece admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:26.514157 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=87,keystone:DELETE=10,keystone:INSERT=3 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:26.526525 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1105/2178] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:13:26 2022] POST /identity/v3/users => generated 292 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:13:26.530625 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d657f82-9944-4317-ae48-a38e8a1b5531 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:26.540777 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d657f82-9944-4317-ae48-a38e8a1b5531 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:26.545356 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bdf3b55-4dd7-4c8c-babb-85a1328ad9e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:26.545939 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bdf3b55-4dd7-4c8c-babb-85a1328ad9e7 admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:26.546218 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bdf3b55-4dd7-4c8c-babb-85a1328ad9e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:26.546490 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bdf3b55-4dd7-4c8c-babb-85a1328ad9e7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:26.547164 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bdf3b55-4dd7-4c8c-babb-85a1328ad9e7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:26.548757 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bdf3b55-4dd7-4c8c-babb-85a1328ad9e7 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:26.562624 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1074/2179] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:13:26 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:13:26.563861 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35e362c4-c36b-4309-9497-1f090c508ece admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:26.575022 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35e362c4-c36b-4309-9497-1f090c508ece admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:26.580169 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1676b6e3-5e85-499b-bebd-bf33c919d947 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:26.580793 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1676b6e3-5e85-499b-bebd-bf33c919d947 admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:26.581071 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1676b6e3-5e85-499b-bebd-bf33c919d947 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:26.581344 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1676b6e3-5e85-499b-bebd-bf33c919d947 admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:26.582005 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1676b6e3-5e85-499b-bebd-bf33c919d947 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:26.583653 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1676b6e3-5e85-499b-bebd-bf33c919d947 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:26.595208 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1106/2180] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:13:26 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 05:13:26.599292 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bdf3b55-4dd7-4c8c-babb-85a1328ad9e7 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:26.609345 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bdf3b55-4dd7-4c8c-babb-85a1328ad9e7 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:26.614706 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdd65f85-b8bc-4923-a2cd-f5cddbdcbc4a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:26.615464 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdd65f85-b8bc-4923-a2cd-f5cddbdcbc4a admin admin] REQUEST_METHOD: `PUT` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:26.615824 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdd65f85-b8bc-4923-a2cd-f5cddbdcbc4a admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:26.616325 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdd65f85-b8bc-4923-a2cd-f5cddbdcbc4a admin admin] PATH_INFO: `/v3/projects/a0799cb263174a40aad16bbb2dd55dd2/users/bc0e4da6b953429caafa307416182aff/roles/6f03d86980c34c36b44a8d6764ff4fda` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:26.633137 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdd65f85-b8bc-4923-a2cd-f5cddbdcbc4a admin admin] RBAC: Authorizing `identity:create_grant(project_id=a0799cb263174a40aad16bbb2dd55dd2, user_id=bc0e4da6b953429caafa307416182aff, role_id=6f03d86980c34c36b44a8d6764ff4fda)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:26.636909 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdd65f85-b8bc-4923-a2cd-f5cddbdcbc4a admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:26.649433 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1075/2181] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:13:26 2022] PUT /identity/v3/projects/a0799cb263174a40aad16bbb2dd55dd2/users/bc0e4da6b953429caafa307416182aff/roles/6f03d86980c34c36b44a8d6764ff4fda => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:13:26.660793 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-647ab6a1-8a05-47b2-9d42-b96d8618039b None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:26.661080 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-647ab6a1-8a05-47b2-9d42-b96d8618039b None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:26.661351 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-647ab6a1-8a05-47b2-9d42-b96d8618039b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:26.696565 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-647ab6a1-8a05-47b2-9d42-b96d8618039b None None] MFA Rules not processed for user `bc0e4da6b953429caafa307416182aff`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:13:26.698068 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-647ab6a1-8a05-47b2-9d42-b96d8618039b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:26.765135 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1107/2182] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:13:26 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:13:26.776246 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdd65f85-b8bc-4923-a2cd-f5cddbdcbc4a admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:26.777942 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-cdd65f85-b8bc-4923-a2cd-f5cddbdcbc4a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:26.831006 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:26.838146 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdd65f85-b8bc-4923-a2cd-f5cddbdcbc4a admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:26.842223 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ed542d4-254f-4efe-8b80-d91765991e1e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '019a79b2affe478b8a30d1c538d0415c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:26.842847 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ed542d4-254f-4efe-8b80-d91765991e1e service cinder] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:26.843252 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ed542d4-254f-4efe-8b80-d91765991e1e service cinder] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:26.843503 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ed542d4-254f-4efe-8b80-d91765991e1e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:26.853784 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ed542d4-254f-4efe-8b80-d91765991e1e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:26.855353 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ed542d4-254f-4efe-8b80-d91765991e1e service cinder] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:26.865681 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1076/2183] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:13:26 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:13:26.884658 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-647ab6a1-8a05-47b2-9d42-b96d8618039b None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:26.955008 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-647ab6a1-8a05-47b2-9d42-b96d8618039b None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:26.959942 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43af4bcb-3f63-4268-90f5-f49dcf0796ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:26.960653 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43af4bcb-3f63-4268-90f5-f49dcf0796ab admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:26.960937 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43af4bcb-3f63-4268-90f5-f49dcf0796ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:26.961238 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43af4bcb-3f63-4268-90f5-f49dcf0796ab admin admin] PATH_INFO: `/v3/users/bc0e4da6b953429caafa307416182aff` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:26.962633 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43af4bcb-3f63-4268-90f5-f49dcf0796ab admin admin] RBAC: Authorizing `identity:delete_user(user_id=bc0e4da6b953429caafa307416182aff)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:26.964614 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43af4bcb-3f63-4268-90f5-f49dcf0796ab admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:27.017965 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-43af4bcb-3f63-4268-90f5-f49dcf0796ab admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bc0e4da6b953429caafa307416182aff'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:27.027645 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-43af4bcb-3f63-4268-90f5-f49dcf0796ab admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bc0e4da6b953429caafa307416182aff'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:27.043860 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-43af4bcb-3f63-4268-90f5-f49dcf0796ab admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bc0e4da6b953429caafa307416182aff'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:27.045881 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:INSERT=2,keystone:SELECT=88,keystone:DELETE=10 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:27.054078 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-43af4bcb-3f63-4268-90f5-f49dcf0796ab admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bc0e4da6b953429caafa307416182aff'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:27.055570 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-43af4bcb-3f63-4268-90f5-f49dcf0796ab admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bc0e4da6b953429caafa307416182aff'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:27.064361 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1108/2184] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:13:26 2022] DELETE /identity/v3/users/bc0e4da6b953429caafa307416182aff => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:13:27.069245 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0f5be73-9b08-45d6-b478-173f532b6860 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:27.069755 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0f5be73-9b08-45d6-b478-173f532b6860 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:27.069877 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0f5be73-9b08-45d6-b478-173f532b6860 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:27.089130 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-c0f5be73-9b08-45d6-b478-173f532b6860 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:13:27.090241 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-c0f5be73-9b08-45d6-b478-173f532b6860 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:27.150781 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1077/2185] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:13:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:13:27.161050 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43af4bcb-3f63-4268-90f5-f49dcf0796ab admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:27.162457 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-43af4bcb-3f63-4268-90f5-f49dcf0796ab admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:27.220187 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0f5be73-9b08-45d6-b478-173f532b6860 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:27.222225 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-c0f5be73-9b08-45d6-b478-173f532b6860 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:27.225951 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43af4bcb-3f63-4268-90f5-f49dcf0796ab admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:27.230524 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-573a4505-ff18-4185-9141-0296ab12ace9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:27.231822 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-573a4505-ff18-4185-9141-0296ab12ace9 service neutron] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:27.232101 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-573a4505-ff18-4185-9141-0296ab12ace9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:27.232372 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-573a4505-ff18-4185-9141-0296ab12ace9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:27.235021 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0f5be73-9b08-45d6-b478-173f532b6860 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:27.239771 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f8914e2-7d75-4b56-a930-0bea8a5681e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:27.240377 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f8914e2-7d75-4b56-a930-0bea8a5681e6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:27.240647 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f8914e2-7d75-4b56-a930-0bea8a5681e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:27.240915 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f8914e2-7d75-4b56-a930-0bea8a5681e6 admin admin] PATH_INFO: `/v3/users/05cb928547c94c74a7e89c39b0cc1fcf` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:27.241728 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-573a4505-ff18-4185-9141-0296ab12ace9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:27.242016 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f8914e2-7d75-4b56-a930-0bea8a5681e6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=05cb928547c94c74a7e89c39b0cc1fcf)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:27.243221 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-573a4505-ff18-4185-9141-0296ab12ace9 service neutron] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:27.244010 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f8914e2-7d75-4b56-a930-0bea8a5681e6 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:27.252762 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1109/2186] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:13:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:13:27.293112 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-8f8914e2-7d75-4b56-a930-0bea8a5681e6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '05cb928547c94c74a7e89c39b0cc1fcf'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:27.293779 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-8f8914e2-7d75-4b56-a930-0bea8a5681e6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '05cb928547c94c74a7e89c39b0cc1fcf'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:27.294466 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-8f8914e2-7d75-4b56-a930-0bea8a5681e6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '05cb928547c94c74a7e89c39b0cc1fcf'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:27.306780 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-8f8914e2-7d75-4b56-a930-0bea8a5681e6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '05cb928547c94c74a7e89c39b0cc1fcf'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:27.314035 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-8f8914e2-7d75-4b56-a930-0bea8a5681e6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '05cb928547c94c74a7e89c39b0cc1fcf'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:27.327623 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1078/2187] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:13:27 2022] DELETE /identity/v3/users/05cb928547c94c74a7e89c39b0cc1fcf => generated 0 bytes in 108 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:13:27.332585 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d8026b-4048-4af4-9e66-70fdc2ca4171 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:27.332867 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d8026b-4048-4af4-9e66-70fdc2ca4171 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:27.333136 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d8026b-4048-4af4-9e66-70fdc2ca4171 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:27.353654 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-e1d8026b-4048-4af4-9e66-70fdc2ca4171 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:13:27.355212 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-e1d8026b-4048-4af4-9e66-70fdc2ca4171 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:27.423501 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1110/2188] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:13:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:13:27.432978 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f8914e2-7d75-4b56-a930-0bea8a5681e6 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:27.435862 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-8f8914e2-7d75-4b56-a930-0bea8a5681e6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:27.501500 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f8914e2-7d75-4b56-a930-0bea8a5681e6 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:27.506350 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f81ea10-09cb-45bc-ae69-5faa8dc05ac5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:27.506996 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f81ea10-09cb-45bc-ae69-5faa8dc05ac5 service neutron] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:27.507277 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f81ea10-09cb-45bc-ae69-5faa8dc05ac5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:27.507550 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f81ea10-09cb-45bc-ae69-5faa8dc05ac5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:27.516827 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f81ea10-09cb-45bc-ae69-5faa8dc05ac5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:27.518868 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f81ea10-09cb-45bc-ae69-5faa8dc05ac5 service neutron] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:27.528728 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1079/2189] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:13:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 05:13:27.620726 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1d8026b-4048-4af4-9e66-70fdc2ca4171 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:27.622516 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-e1d8026b-4048-4af4-9e66-70fdc2ca4171 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:27.640383 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1d8026b-4048-4af4-9e66-70fdc2ca4171 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:27.647368 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d449af6-9ecc-446d-9fd8-f77f0080fcf9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:27.648012 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d449af6-9ecc-446d-9fd8-f77f0080fcf9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:27.648289 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d449af6-9ecc-446d-9fd8-f77f0080fcf9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:27.648558 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d449af6-9ecc-446d-9fd8-f77f0080fcf9 admin admin] PATH_INFO: `/v3/projects/a0799cb263174a40aad16bbb2dd55dd2` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:27.649702 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d449af6-9ecc-446d-9fd8-f77f0080fcf9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a0799cb263174a40aad16bbb2dd55dd2)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:27.651661 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d449af6-9ecc-446d-9fd8-f77f0080fcf9 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:27.707549 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-7d449af6-9ecc-446d-9fd8-f77f0080fcf9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a0799cb263174a40aad16bbb2dd55dd2', 'request_id': 'req-7d449af6-9ecc-446d-9fd8-f77f0080fcf9'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:27.716525 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-7d449af6-9ecc-446d-9fd8-f77f0080fcf9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a0799cb263174a40aad16bbb2dd55dd2', 'request_id': 'req-7d449af6-9ecc-446d-9fd8-f77f0080fcf9'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:27.723502 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-7d449af6-9ecc-446d-9fd8-f77f0080fcf9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a0799cb263174a40aad16bbb2dd55dd2', 'request_id': 'req-7d449af6-9ecc-446d-9fd8-f77f0080fcf9'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:27.733844 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-7d449af6-9ecc-446d-9fd8-f77f0080fcf9 admin admin] The token cache is being invalidate because project a0799cb263174a40aad16bbb2dd55dd2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81625) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:13:27.734162 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-7d449af6-9ecc-446d-9fd8-f77f0080fcf9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:27.734783 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-7d449af6-9ecc-446d-9fd8-f77f0080fcf9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:27.737006 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1111/2190] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:13:27 2022] DELETE /identity/v3/projects/a0799cb263174a40aad16bbb2dd55dd2 => generated 0 bytes in 117 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:13:27.742024 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a1d515-778e-4f20-8341-16f4601e4e4f None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:27.742312 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a1d515-778e-4f20-8341-16f4601e4e4f None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:27.742584 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a1d515-778e-4f20-8341-16f4601e4e4f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:27.754475 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:27.768605 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-13a1d515-778e-4f20-8341-16f4601e4e4f None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:13:27.772998 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-13a1d515-778e-4f20-8341-16f4601e4e4f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:27.774121 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d449af6-9ecc-446d-9fd8-f77f0080fcf9 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:27.775860 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-7d449af6-9ecc-446d-9fd8-f77f0080fcf9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:27.800819 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:27.842074 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1080/2191] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:13:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:13:27.849836 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13a1d515-778e-4f20-8341-16f4601e4e4f None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:27.851247 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-13a1d515-778e-4f20-8341-16f4601e4e4f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:27.855667 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d449af6-9ecc-446d-9fd8-f77f0080fcf9 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:27.862585 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f273b49-f7ec-4961-87da-e73c520bd813 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:27.863441 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f273b49-f7ec-4961-87da-e73c520bd813 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:27.864070 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13a1d515-778e-4f20-8341-16f4601e4e4f None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:27.867132 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f273b49-f7ec-4961-87da-e73c520bd813 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:27.867494 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f273b49-f7ec-4961-87da-e73c520bd813 admin admin] PATH_INFO: `/v3/users/f4322a87e0d441608afa20c69992eb3a` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:27.868715 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81570c98-047a-40c2-ae35-ac669ce6a2d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:27.868898 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f273b49-f7ec-4961-87da-e73c520bd813 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f4322a87e0d441608afa20c69992eb3a)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:27.869369 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81570c98-047a-40c2-ae35-ac669ce6a2d6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:27.869653 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81570c98-047a-40c2-ae35-ac669ce6a2d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:27.869928 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81570c98-047a-40c2-ae35-ac669ce6a2d6 admin admin] PATH_INFO: `/v3/projects/2b930d0a13b84f7baa5012fc1d686b4c` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:27.871071 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81570c98-047a-40c2-ae35-ac669ce6a2d6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2b930d0a13b84f7baa5012fc1d686b4c)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:27.871221 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f273b49-f7ec-4961-87da-e73c520bd813 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:27.872967 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81570c98-047a-40c2-ae35-ac669ce6a2d6 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:27.922487 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-81570c98-047a-40c2-ae35-ac669ce6a2d6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2b930d0a13b84f7baa5012fc1d686b4c', 'request_id': 'req-81570c98-047a-40c2-ae35-ac669ce6a2d6'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:27.922687 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-0f273b49-f7ec-4961-87da-e73c520bd813 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f4322a87e0d441608afa20c69992eb3a'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:27.930481 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-81570c98-047a-40c2-ae35-ac669ce6a2d6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2b930d0a13b84f7baa5012fc1d686b4c', 'request_id': 'req-81570c98-047a-40c2-ae35-ac669ce6a2d6'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:27.933225 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-0f273b49-f7ec-4961-87da-e73c520bd813 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f4322a87e0d441608afa20c69992eb3a'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:27.938081 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-81570c98-047a-40c2-ae35-ac669ce6a2d6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2b930d0a13b84f7baa5012fc1d686b4c', 'request_id': 'req-81570c98-047a-40c2-ae35-ac669ce6a2d6'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:27.946581 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-0f273b49-f7ec-4961-87da-e73c520bd813 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f4322a87e0d441608afa20c69992eb3a'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:27.948976 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-81570c98-047a-40c2-ae35-ac669ce6a2d6 admin admin] The token cache is being invalidate because project 2b930d0a13b84f7baa5012fc1d686b4c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81624) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:13:27.949261 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-81570c98-047a-40c2-ae35-ac669ce6a2d6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:27.949908 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-81570c98-047a-40c2-ae35-ac669ce6a2d6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:27.951414 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1081/2192] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:13:27 2022] DELETE /identity/v3/projects/2b930d0a13b84f7baa5012fc1d686b4c => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:13:27.953058 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e25dbfb8-5bcb-485d-8b97-e895100cac6e None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:27.953345 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e25dbfb8-5bcb-485d-8b97-e895100cac6e None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:27.953619 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e25dbfb8-5bcb-485d-8b97-e895100cac6e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:27.954203 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-0f273b49-f7ec-4961-87da-e73c520bd813 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f4322a87e0d441608afa20c69992eb3a'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:27.954934 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-0f273b49-f7ec-4961-87da-e73c520bd813 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f4322a87e0d441608afa20c69992eb3a'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:27.961198 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1112/2193] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:13:27 2022] DELETE /identity/v3/users/f4322a87e0d441608afa20c69992eb3a => generated 0 bytes in 188 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:13:27.962588 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3066dfd-81aa-4b85-a498-a296a7980525 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:27.962990 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3066dfd-81aa-4b85-a498-a296a7980525 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:27.963360 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3066dfd-81aa-4b85-a498-a296a7980525 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:27.978056 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-e25dbfb8-5bcb-485d-8b97-e895100cac6e None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:13:27.980231 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-e25dbfb8-5bcb-485d-8b97-e895100cac6e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:27.983951 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-d3066dfd-81aa-4b85-a498-a296a7980525 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:13:27.985137 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-d3066dfd-81aa-4b85-a498-a296a7980525 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:28.040796 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1082/2194] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:13:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:13:28.042041 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbc50ce6-ade1-4404-b109-d6d16f7a1015 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:28.042329 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbc50ce6-ade1-4404-b109-d6d16f7a1015 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:28.042603 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbc50ce6-ade1-4404-b109-d6d16f7a1015 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:28.049821 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1113/2195] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:13:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:13:28.050731 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3066dfd-81aa-4b85-a498-a296a7980525 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:28.060580 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3066dfd-81aa-4b85-a498-a296a7980525 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:28.065849 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cca57c19-3654-4c4f-b618-ff9576e5bc1b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:28.066433 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca57c19-3654-4c4f-b618-ff9576e5bc1b admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:28.066708 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca57c19-3654-4c4f-b618-ff9576e5bc1b admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:28.067010 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca57c19-3654-4c4f-b618-ff9576e5bc1b admin admin] PATH_INFO: `/v3/domains` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:28.067642 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cca57c19-3654-4c4f-b618-ff9576e5bc1b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:13:28.067940 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cca57c19-3654-4c4f-b618-ff9576e5bc1b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:28.068382 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-fbc50ce6-ade1-4404-b109-d6d16f7a1015 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:13:28.069518 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cca57c19-3654-4c4f-b618-ff9576e5bc1b admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:28.069639 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-fbc50ce6-ade1-4404-b109-d6d16f7a1015 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:28.077421 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1083/2196] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:13:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:13:28.078912 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b9b1e54-f477-44b2-a0cd-dfa4783207ed None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:28.079195 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b9b1e54-f477-44b2-a0cd-dfa4783207ed None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:28.079471 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b9b1e54-f477-44b2-a0cd-dfa4783207ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:28.087877 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1114/2197] 10.222.0.77 () {58 vars in 1153 bytes} [Tue Aug 30 05:13:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:13:28.088848 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cca57c19-3654-4c4f-b618-ff9576e5bc1b admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:28.090500 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-cca57c19-3654-4c4f-b618-ff9576e5bc1b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:28.100329 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-4b9b1e54-f477-44b2-a0cd-dfa4783207ed None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:13:28.106772 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-4b9b1e54-f477-44b2-a0cd-dfa4783207ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:28.109712 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1084/2198] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:13:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:13:28.110607 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b9b1e54-f477-44b2-a0cd-dfa4783207ed None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:28.120742 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b9b1e54-f477-44b2-a0cd-dfa4783207ed None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:28.125457 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06eb322a-d334-450e-8a54-ff525ee89750 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:28.126214 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06eb322a-d334-450e-8a54-ff525ee89750 admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:28.126324 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06eb322a-d334-450e-8a54-ff525ee89750 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:28.126583 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06eb322a-d334-450e-8a54-ff525ee89750 admin admin] PATH_INFO: `/v3/projects` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:28.127927 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06eb322a-d334-450e-8a54-ff525ee89750 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:28.128942 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:28.129872 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06eb322a-d334-450e-8a54-ff525ee89750 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:28.147158 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:28.164611 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1085/2199] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:13:28 2022] POST /identity/v3/projects => generated 385 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:13:28.165653 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cca57c19-3654-4c4f-b618-ff9576e5bc1b admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:28.165782 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06eb322a-d334-450e-8a54-ff525ee89750 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:28.223412 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c297331d-8721-43f9-b8e3-04bbb3a43e23 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:28.224018 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c297331d-8721-43f9-b8e3-04bbb3a43e23 service neutron] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:28.224298 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c297331d-8721-43f9-b8e3-04bbb3a43e23 service neutron] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:28.224568 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c297331d-8721-43f9-b8e3-04bbb3a43e23 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:28.246334 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06eb322a-d334-450e-8a54-ff525ee89750 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:28.252406 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-518a0761-3154-47d7-866b-92471423d1c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:28.252668 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-518a0761-3154-47d7-866b-92471423d1c5 admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:28.252856 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-518a0761-3154-47d7-866b-92471423d1c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:28.253090 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-518a0761-3154-47d7-866b-92471423d1c5 admin admin] PATH_INFO: `/v3/domains` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:28.253923 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-518a0761-3154-47d7-866b-92471423d1c5 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:13:28.254590 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-518a0761-3154-47d7-866b-92471423d1c5 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:28.256428 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-518a0761-3154-47d7-866b-92471423d1c5 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:28.274868 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1086/2200] 10.222.0.77 () {58 vars in 1153 bytes} [Tue Aug 30 05:13:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 110 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:13:28.276849 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-518a0761-3154-47d7-866b-92471423d1c5 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:28.301211 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-518a0761-3154-47d7-866b-92471423d1c5 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:28.306056 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6a21a6b-e44c-4685-b1c6-89794f097325 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:28.306737 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6a21a6b-e44c-4685-b1c6-89794f097325 admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:28.307226 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6a21a6b-e44c-4685-b1c6-89794f097325 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:28.307571 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6a21a6b-e44c-4685-b1c6-89794f097325 admin admin] PATH_INFO: `/v3/users` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:28.308263 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6a21a6b-e44c-4685-b1c6-89794f097325 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:28.310361 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6a21a6b-e44c-4685-b1c6-89794f097325 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:28.312900 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c297331d-8721-43f9-b8e3-04bbb3a43e23 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:28.314680 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c297331d-8721-43f9-b8e3-04bbb3a43e23 service neutron] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:28.326222 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1115/2201] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:13:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 238 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:13:28.327048 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c297331d-8721-43f9-b8e3-04bbb3a43e23 service neutron] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:28.338144 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c297331d-8721-43f9-b8e3-04bbb3a43e23 service neutron] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:28.340735 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1087/2202] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:13:28 2022] POST /identity/v3/users => generated 293 bytes in 64 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:13:28.345171 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6a21a6b-e44c-4685-b1c6-89794f097325 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:28.348151 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8509fe81-2ec8-4afd-9e2b-1eecfb6ce3f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:28.348874 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8509fe81-2ec8-4afd-9e2b-1eecfb6ce3f6 admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:28.349238 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8509fe81-2ec8-4afd-9e2b-1eecfb6ce3f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:28.349564 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8509fe81-2ec8-4afd-9e2b-1eecfb6ce3f6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:28.350256 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8509fe81-2ec8-4afd-9e2b-1eecfb6ce3f6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:28.353570 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8509fe81-2ec8-4afd-9e2b-1eecfb6ce3f6 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:28.356711 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6a21a6b-e44c-4685-b1c6-89794f097325 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:28.362202 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1512d226-c08a-4206-9174-2a8ee6164e58 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:28.362733 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1512d226-c08a-4206-9174-2a8ee6164e58 admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:28.363058 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1512d226-c08a-4206-9174-2a8ee6164e58 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:28.363472 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1512d226-c08a-4206-9174-2a8ee6164e58 admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:28.364117 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1512d226-c08a-4206-9174-2a8ee6164e58 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:28.365761 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1512d226-c08a-4206-9174-2a8ee6164e58 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:28.380615 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1088/2203] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:13:28 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 05:13:28.384650 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1512d226-c08a-4206-9174-2a8ee6164e58 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:28.392114 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1116/2204] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:13:28 2022] POST /identity/v3/projects => generated 373 bytes in 62 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:13:28.393679 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8509fe81-2ec8-4afd-9e2b-1eecfb6ce3f6 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:28.396054 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1512d226-c08a-4206-9174-2a8ee6164e58 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:28.473156 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d21b9c9-7204-4685-b497-1b27adc31043 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:28.473771 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d21b9c9-7204-4685-b497-1b27adc31043 admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:28.474048 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d21b9c9-7204-4685-b497-1b27adc31043 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:28.474314 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d21b9c9-7204-4685-b497-1b27adc31043 admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:28.474983 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d21b9c9-7204-4685-b497-1b27adc31043 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:28.477336 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d21b9c9-7204-4685-b497-1b27adc31043 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:28.485096 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8509fe81-2ec8-4afd-9e2b-1eecfb6ce3f6 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:28.492472 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1089/2205] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:13:28 2022] GET /identity/v3/roles => generated 1420 bytes in 108 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 05:13:28.494786 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-261f86af-8eec-4f1c-b5d1-ceb3ca7f3224 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:28.495475 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-261f86af-8eec-4f1c-b5d1-ceb3ca7f3224 admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:28.495756 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-261f86af-8eec-4f1c-b5d1-ceb3ca7f3224 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:28.496032 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-261f86af-8eec-4f1c-b5d1-ceb3ca7f3224 admin admin] PATH_INFO: `/v3/users` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:28.496672 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-261f86af-8eec-4f1c-b5d1-ceb3ca7f3224 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:28.497210 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d21b9c9-7204-4685-b497-1b27adc31043 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:28.498970 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-261f86af-8eec-4f1c-b5d1-ceb3ca7f3224 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:28.508419 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d21b9c9-7204-4685-b497-1b27adc31043 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:28.514004 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc10a48b-ed2c-4eb6-9742-ce5216412c28 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:28.514402 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc10a48b-ed2c-4eb6-9742-ce5216412c28 admin admin] REQUEST_METHOD: `PUT` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:28.514730 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc10a48b-ed2c-4eb6-9742-ce5216412c28 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:28.515061 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc10a48b-ed2c-4eb6-9742-ce5216412c28 admin admin] PATH_INFO: `/v3/projects/1d6fb13dad924bdaa2588b516fbc9165/users/9b3c8e44ba2b42bd8a61d1823e8a2744/roles/6f03d86980c34c36b44a8d6764ff4fda` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:28.520612 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:28.532053 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1117/2206] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:13:28 2022] POST /identity/v3/users => generated 287 bytes in 136 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 05:13:28.536308 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-261f86af-8eec-4f1c-b5d1-ceb3ca7f3224 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:28.547602 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc10a48b-ed2c-4eb6-9742-ce5216412c28 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1d6fb13dad924bdaa2588b516fbc9165, user_id=9b3c8e44ba2b42bd8a61d1823e8a2744, role_id=6f03d86980c34c36b44a8d6764ff4fda)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:28.559292 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc10a48b-ed2c-4eb6-9742-ce5216412c28 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:28.561678 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-261f86af-8eec-4f1c-b5d1-ceb3ca7f3224 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:28.567000 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fdc40ce-50c1-4952-aa20-2f8b2bd856d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:28.567919 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fdc40ce-50c1-4952-aa20-2f8b2bd856d9 admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:28.567919 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fdc40ce-50c1-4952-aa20-2f8b2bd856d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:28.568244 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fdc40ce-50c1-4952-aa20-2f8b2bd856d9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:28.569082 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fdc40ce-50c1-4952-aa20-2f8b2bd856d9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:28.570879 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fdc40ce-50c1-4952-aa20-2f8b2bd856d9 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:28.584397 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1090/2207] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:13:28 2022] PUT /identity/v3/projects/1d6fb13dad924bdaa2588b516fbc9165/users/9b3c8e44ba2b42bd8a61d1823e8a2744/roles/6f03d86980c34c36b44a8d6764ff4fda => generated 0 bytes in 88 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 05:13:28.585085 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1118/2208] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:13:28 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 05:13:28.588924 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc10a48b-ed2c-4eb6-9742-ce5216412c28 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:28.595504 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b83e1d7-3c2a-418d-872c-7e853c2e1b14 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:28.596509 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b83e1d7-3c2a-418d-872c-7e853c2e1b14 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:28.597170 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b83e1d7-3c2a-418d-872c-7e853c2e1b14 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:28.636304 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:INSERT=5,keystone:SELECT=95 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:28.659313 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-2b83e1d7-3c2a-418d-872c-7e853c2e1b14 None None] MFA Rules not processed for user `9b3c8e44ba2b42bd8a61d1823e8a2744`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:13:28.660587 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-2b83e1d7-3c2a-418d-872c-7e853c2e1b14 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:28.688003 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc10a48b-ed2c-4eb6-9742-ce5216412c28 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:28.706762 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1de9ac3-e21a-48ff-97c6-60850dfc9214 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:28.707484 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1de9ac3-e21a-48ff-97c6-60850dfc9214 admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:28.707773 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1de9ac3-e21a-48ff-97c6-60850dfc9214 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:28.708045 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1de9ac3-e21a-48ff-97c6-60850dfc9214 admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:28.708762 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1de9ac3-e21a-48ff-97c6-60850dfc9214 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:28.710355 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1de9ac3-e21a-48ff-97c6-60850dfc9214 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:28.734725 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1091/2209] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:13:28 2022] GET /identity/v3/roles => generated 1420 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 05:13:28.735355 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1de9ac3-e21a-48ff-97c6-60850dfc9214 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:28.739067 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-b1de9ac3-e21a-48ff-97c6-60850dfc9214 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:28.754149 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1de9ac3-e21a-48ff-97c6-60850dfc9214 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:28.759417 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81d7d927-5431-4e1b-8cf2-b17c9bfefd85 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:28.760045 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81d7d927-5431-4e1b-8cf2-b17c9bfefd85 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:28.760399 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81d7d927-5431-4e1b-8cf2-b17c9bfefd85 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:28.760738 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81d7d927-5431-4e1b-8cf2-b17c9bfefd85 admin admin] PATH_INFO: `/v3/projects/e656eb9012c441398c8ad5bdcc6c4046` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:28.761902 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81d7d927-5431-4e1b-8cf2-b17c9bfefd85 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e656eb9012c441398c8ad5bdcc6c4046)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:28.762525 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1119/2210] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:13:28 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 05:13:28.763578 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b83e1d7-3c2a-418d-872c-7e853c2e1b14 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:28.763938 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81d7d927-5431-4e1b-8cf2-b17c9bfefd85 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:28.775834 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b83e1d7-3c2a-418d-872c-7e853c2e1b14 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:28.783293 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03804211-888f-402d-9b51-fde5028e511d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:28.784446 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03804211-888f-402d-9b51-fde5028e511d admin admin] REQUEST_METHOD: `PUT` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:28.784750 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03804211-888f-402d-9b51-fde5028e511d admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:28.784993 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03804211-888f-402d-9b51-fde5028e511d admin admin] PATH_INFO: `/v3/projects/5ba05d601a9240bb922f4d6dc3cc6bc6/users/71e9c605498544c8b6c75349d9b66a94/roles/6f03d86980c34c36b44a8d6764ff4fda` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:28.803221 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03804211-888f-402d-9b51-fde5028e511d admin admin] RBAC: Authorizing `identity:create_grant(project_id=5ba05d601a9240bb922f4d6dc3cc6bc6, user_id=71e9c605498544c8b6c75349d9b66a94, role_id=6f03d86980c34c36b44a8d6764ff4fda)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:28.806280 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03804211-888f-402d-9b51-fde5028e511d admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:28.825406 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1120/2211] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:13:28 2022] PUT /identity/v3/projects/5ba05d601a9240bb922f4d6dc3cc6bc6/users/71e9c605498544c8b6c75349d9b66a94/roles/6f03d86980c34c36b44a8d6764ff4fda => generated 0 bytes in 62 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:13:28.826323 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03804211-888f-402d-9b51-fde5028e511d admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:28.828134 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-03804211-888f-402d-9b51-fde5028e511d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:28.829496 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-81d7d927-5431-4e1b-8cf2-b17c9bfefd85 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e656eb9012c441398c8ad5bdcc6c4046', 'request_id': 'req-81d7d927-5431-4e1b-8cf2-b17c9bfefd85'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:28.841363 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-81d7d927-5431-4e1b-8cf2-b17c9bfefd85 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e656eb9012c441398c8ad5bdcc6c4046', 'request_id': 'req-81d7d927-5431-4e1b-8cf2-b17c9bfefd85'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:28.867665 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-81d7d927-5431-4e1b-8cf2-b17c9bfefd85 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e656eb9012c441398c8ad5bdcc6c4046', 'request_id': 'req-81d7d927-5431-4e1b-8cf2-b17c9bfefd85'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:28.869428 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-81d7d927-5431-4e1b-8cf2-b17c9bfefd85 admin admin] The token cache is being invalidate because project e656eb9012c441398c8ad5bdcc6c4046 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81624) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:13:28.869732 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-81d7d927-5431-4e1b-8cf2-b17c9bfefd85 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:28.870536 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-81d7d927-5431-4e1b-8cf2-b17c9bfefd85 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:28.872592 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1092/2212] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:13:28 2022] DELETE /identity/v3/projects/e656eb9012c441398c8ad5bdcc6c4046 => generated 0 bytes in 138 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:13:28.874699 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82123fb6-6b00-4b89-bae2-0558a77c8ec6 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:28.875151 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82123fb6-6b00-4b89-bae2-0558a77c8ec6 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:28.875919 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82123fb6-6b00-4b89-bae2-0558a77c8ec6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:28.904722 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03804211-888f-402d-9b51-fde5028e511d admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:28.906911 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-82123fb6-6b00-4b89-bae2-0558a77c8ec6 None None] MFA Rules not processed for user `71e9c605498544c8b6c75349d9b66a94`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:13:28.908138 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-82123fb6-6b00-4b89-bae2-0558a77c8ec6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:28.909725 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdc6dd6a-ad85-4275-b834-f2df31ffd5ed service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '019a79b2affe478b8a30d1c538d0415c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:28.910443 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdc6dd6a-ad85-4275-b834-f2df31ffd5ed service cinder] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:28.911353 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdc6dd6a-ad85-4275-b834-f2df31ffd5ed service cinder] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:28.911631 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdc6dd6a-ad85-4275-b834-f2df31ffd5ed service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:28.913477 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-cdc6dd6a-ad85-4275-b834-f2df31ffd5ed service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:28.983085 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdc6dd6a-ad85-4275-b834-f2df31ffd5ed service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:28.984841 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdc6dd6a-ad85-4275-b834-f2df31ffd5ed service cinder] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:28.985741 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1093/2213] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:13:28 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:13:28.987399 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b9f4ea4-0bab-49fd-8890-4c526f51a8d3 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:28.987691 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b9f4ea4-0bab-49fd-8890-4c526f51a8d3 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:28.987973 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b9f4ea4-0bab-49fd-8890-4c526f51a8d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:29.001049 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1121/2214] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:13:28 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 175 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:13:29.002403 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdc6dd6a-ad85-4275-b834-f2df31ffd5ed service cinder] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:29.003762 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-cdc6dd6a-ad85-4275-b834-f2df31ffd5ed service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:29.010483 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-5b9f4ea4-0bab-49fd-8890-4c526f51a8d3 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:13:29.011761 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-5b9f4ea4-0bab-49fd-8890-4c526f51a8d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:29.022617 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdc6dd6a-ad85-4275-b834-f2df31ffd5ed service cinder] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:29.029761 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05debae8-246b-4eb2-9de2-5b474c1bfabf service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '019a79b2affe478b8a30d1c538d0415c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:29.029761 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05debae8-246b-4eb2-9de2-5b474c1bfabf service cinder] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:29.029761 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05debae8-246b-4eb2-9de2-5b474c1bfabf service cinder] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:29.029761 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05debae8-246b-4eb2-9de2-5b474c1bfabf service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:29.036318 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:29.042807 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05debae8-246b-4eb2-9de2-5b474c1bfabf service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:29.044597 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05debae8-246b-4eb2-9de2-5b474c1bfabf service cinder] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:29.060610 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1122/2215] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:13:29 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 59 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:13:29.065823 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:29.102083 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1094/2216] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:13:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:13:29.111848 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a878eb5c-102c-4353-8e38-b784cc0e5036 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:29.112142 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a878eb5c-102c-4353-8e38-b784cc0e5036 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:29.112403 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a878eb5c-102c-4353-8e38-b784cc0e5036 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:29.140495 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-a878eb5c-102c-4353-8e38-b784cc0e5036 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:13:29.141205 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-a878eb5c-102c-4353-8e38-b784cc0e5036 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:29.151987 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1123/2217] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:13:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:13:29.156328 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b9f4ea4-0bab-49fd-8890-4c526f51a8d3 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:29.166472 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b9f4ea4-0bab-49fd-8890-4c526f51a8d3 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:29.171493 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8469caf0-7d62-4162-8cc9-1ca432932153 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:29.172449 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8469caf0-7d62-4162-8cc9-1ca432932153 admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:29.172449 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8469caf0-7d62-4162-8cc9-1ca432932153 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:29.172669 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8469caf0-7d62-4162-8cc9-1ca432932153 admin admin] PATH_INFO: `/v3/domains` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:29.173269 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8469caf0-7d62-4162-8cc9-1ca432932153 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:13:29.173572 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8469caf0-7d62-4162-8cc9-1ca432932153 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:29.175262 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8469caf0-7d62-4162-8cc9-1ca432932153 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:29.189972 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1095/2218] 10.222.0.77 () {58 vars in 1153 bytes} [Tue Aug 30 05:13:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:13:29.194273 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a878eb5c-102c-4353-8e38-b784cc0e5036 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:29.205875 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a878eb5c-102c-4353-8e38-b784cc0e5036 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:29.210049 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bd827ad-4143-442b-b86e-598c1205eef9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:29.210648 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bd827ad-4143-442b-b86e-598c1205eef9 admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:29.210960 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bd827ad-4143-442b-b86e-598c1205eef9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:29.211217 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bd827ad-4143-442b-b86e-598c1205eef9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:29.211851 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bd827ad-4143-442b-b86e-598c1205eef9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:29.214052 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bd827ad-4143-442b-b86e-598c1205eef9 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:29.250648 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8469caf0-7d62-4162-8cc9-1ca432932153 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:29.260057 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1124/2219] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:13:29 2022] POST /identity/v3/projects => generated 363 bytes in 66 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:13:29.264116 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8469caf0-7d62-4162-8cc9-1ca432932153 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:29.349068 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3cdfcdc-05bd-4354-ab1f-d73b7ffee1fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:29.350016 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3cdfcdc-05bd-4354-ab1f-d73b7ffee1fe admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:29.350329 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3cdfcdc-05bd-4354-ab1f-d73b7ffee1fe admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:29.350610 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3cdfcdc-05bd-4354-ab1f-d73b7ffee1fe admin admin] PATH_INFO: `/v3/users` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:29.351545 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3cdfcdc-05bd-4354-ab1f-d73b7ffee1fe admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:29.353894 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3cdfcdc-05bd-4354-ab1f-d73b7ffee1fe admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:29.382440 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1096/2220] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:13:29 2022] POST /identity/v3/users => generated 282 bytes in 132 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 05:13:29.396419 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bd827ad-4143-442b-b86e-598c1205eef9 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:29.414388 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bd827ad-4143-442b-b86e-598c1205eef9 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:29.419247 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bca2a76-3ce7-4450-8146-f49b2dc9b50d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:29.419846 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bca2a76-3ce7-4450-8146-f49b2dc9b50d admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:29.420122 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bca2a76-3ce7-4450-8146-f49b2dc9b50d admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:29.420388 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bca2a76-3ce7-4450-8146-f49b2dc9b50d admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:29.421028 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bca2a76-3ce7-4450-8146-f49b2dc9b50d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:29.422807 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bca2a76-3ce7-4450-8146-f49b2dc9b50d admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:29.454978 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1125/2221] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:13:29 2022] GET /identity/v3/roles => generated 1420 bytes in 57 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:13:29.463269 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3cdfcdc-05bd-4354-ab1f-d73b7ffee1fe admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:29.475443 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3cdfcdc-05bd-4354-ab1f-d73b7ffee1fe admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:29.480338 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c988dc0-576b-4027-bc27-2ea0ee6606f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:29.480942 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c988dc0-576b-4027-bc27-2ea0ee6606f8 admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:29.481220 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c988dc0-576b-4027-bc27-2ea0ee6606f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:29.481494 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c988dc0-576b-4027-bc27-2ea0ee6606f8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:29.482133 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c988dc0-576b-4027-bc27-2ea0ee6606f8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:29.483766 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c988dc0-576b-4027-bc27-2ea0ee6606f8 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:29.495416 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1097/2222] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:13:29 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:13:29.499450 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bca2a76-3ce7-4450-8146-f49b2dc9b50d admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:29.512187 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bca2a76-3ce7-4450-8146-f49b2dc9b50d admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:29.517204 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8959774f-f681-469e-8596-92dcc71bf7d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:29.517878 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8959774f-f681-469e-8596-92dcc71bf7d8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:29.518218 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8959774f-f681-469e-8596-92dcc71bf7d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:29.518428 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8959774f-f681-469e-8596-92dcc71bf7d8 admin admin] PATH_INFO: `/v3/projects/8a84d854a2164b2e880984e5f277d574/users/08f2705fc76b4ae192626fb4de9ae6a3/roles/6f03d86980c34c36b44a8d6764ff4fda` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:29.534763 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8959774f-f681-469e-8596-92dcc71bf7d8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8a84d854a2164b2e880984e5f277d574, user_id=08f2705fc76b4ae192626fb4de9ae6a3, role_id=6f03d86980c34c36b44a8d6764ff4fda)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:29.537572 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8959774f-f681-469e-8596-92dcc71bf7d8 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:29.550974 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1126/2223] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:13:29 2022] PUT /identity/v3/projects/8a84d854a2164b2e880984e5f277d574/users/08f2705fc76b4ae192626fb4de9ae6a3/roles/6f03d86980c34c36b44a8d6764ff4fda => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:13:29.562562 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8865fd7c-4444-4bfb-8a9e-f3d4a9eef00e None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:29.562896 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8865fd7c-4444-4bfb-8a9e-f3d4a9eef00e None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:29.563215 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8865fd7c-4444-4bfb-8a9e-f3d4a9eef00e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:29.590991 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-8865fd7c-4444-4bfb-8a9e-f3d4a9eef00e None None] MFA Rules not processed for user `08f2705fc76b4ae192626fb4de9ae6a3`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:13:29.592285 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-8865fd7c-4444-4bfb-8a9e-f3d4a9eef00e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:29.656684 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1098/2224] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:13:29 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 05:13:29.661503 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8959774f-f681-469e-8596-92dcc71bf7d8 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:29.723907 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8959774f-f681-469e-8596-92dcc71bf7d8 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:29.728678 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f19a3e6d-a3b8-4482-9d96-c93bcf37bc2c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:29.729301 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f19a3e6d-a3b8-4482-9d96-c93bcf37bc2c admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:29.729581 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f19a3e6d-a3b8-4482-9d96-c93bcf37bc2c admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:29.729854 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f19a3e6d-a3b8-4482-9d96-c93bcf37bc2c admin admin] PATH_INFO: `/v3/projects` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:29.730485 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f19a3e6d-a3b8-4482-9d96-c93bcf37bc2c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:29.732419 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f19a3e6d-a3b8-4482-9d96-c93bcf37bc2c admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:29.758869 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1127/2225] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:13:29 2022] POST /identity/v3/projects => generated 365 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 05:13:29.762624 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8865fd7c-4444-4bfb-8a9e-f3d4a9eef00e None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:29.773117 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8865fd7c-4444-4bfb-8a9e-f3d4a9eef00e None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:29.805435 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:29.830106 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aee880be-6517-4616-ba47-f263b025e715 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:29.830709 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee880be-6517-4616-ba47-f263b025e715 admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:29.831021 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee880be-6517-4616-ba47-f263b025e715 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:29.831295 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee880be-6517-4616-ba47-f263b025e715 admin admin] PATH_INFO: `/v3/users` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:29.831914 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aee880be-6517-4616-ba47-f263b025e715 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:29.833826 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aee880be-6517-4616-ba47-f263b025e715 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:29.855640 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1099/2226] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:13:29 2022] POST /identity/v3/users => generated 289 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:13:29.859605 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f19a3e6d-a3b8-4482-9d96-c93bcf37bc2c admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:29.870536 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f19a3e6d-a3b8-4482-9d96-c93bcf37bc2c admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:29.875357 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-933ba77f-a6fa-4ffc-a5c2-18670e0c5ca2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:29.876405 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-933ba77f-a6fa-4ffc-a5c2-18670e0c5ca2 admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:29.876896 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-933ba77f-a6fa-4ffc-a5c2-18670e0c5ca2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:29.877288 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-933ba77f-a6fa-4ffc-a5c2-18670e0c5ca2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:29.877993 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-933ba77f-a6fa-4ffc-a5c2-18670e0c5ca2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:29.879966 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-933ba77f-a6fa-4ffc-a5c2-18670e0c5ca2 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:29.896230 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1128/2227] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:13:29 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:13:29.897426 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aee880be-6517-4616-ba47-f263b025e715 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:29.907765 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aee880be-6517-4616-ba47-f263b025e715 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:29.912939 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e6f859-4fed-4703-859f-7453ce623d29 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:29.913584 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e6f859-4fed-4703-859f-7453ce623d29 admin admin] REQUEST_METHOD: `PUT` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:29.913866 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e6f859-4fed-4703-859f-7453ce623d29 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:29.914141 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e6f859-4fed-4703-859f-7453ce623d29 admin admin] PATH_INFO: `/v3/projects/7d2e3b7326644d01ab8000f8575c479b/users/ad5a8a3bea3a44019965d8c93730003c/roles/9d9d5d2c270146488249b7ed943280db` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:29.930168 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18e6f859-4fed-4703-859f-7453ce623d29 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7d2e3b7326644d01ab8000f8575c479b, user_id=ad5a8a3bea3a44019965d8c93730003c, role_id=9d9d5d2c270146488249b7ed943280db)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:29.932987 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18e6f859-4fed-4703-859f-7453ce623d29 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:29.946012 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1100/2228] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:13:29 2022] PUT /identity/v3/projects/7d2e3b7326644d01ab8000f8575c479b/users/ad5a8a3bea3a44019965d8c93730003c/roles/9d9d5d2c270146488249b7ed943280db => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 05:13:29.950077 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-933ba77f-a6fa-4ffc-a5c2-18670e0c5ca2 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:29.961083 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-933ba77f-a6fa-4ffc-a5c2-18670e0c5ca2 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:30.021075 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae4d9b86-b422-4af9-94b0-915ed672734b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:30.021803 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae4d9b86-b422-4af9-94b0-915ed672734b admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:30.022138 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae4d9b86-b422-4af9-94b0-915ed672734b admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:30.022410 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae4d9b86-b422-4af9-94b0-915ed672734b admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:30.023235 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae4d9b86-b422-4af9-94b0-915ed672734b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:30.025130 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae4d9b86-b422-4af9-94b0-915ed672734b admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:30.030756 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:30.042498 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1129/2229] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:13:29 2022] GET /identity/v3/roles => generated 1420 bytes in 93 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:13:30.043479 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e6f859-4fed-4703-859f-7453ce623d29 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:30.054338 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e6f859-4fed-4703-859f-7453ce623d29 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:30.058884 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c859edd-46f0-4734-b37a-71931430e94c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:30.059521 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c859edd-46f0-4734-b37a-71931430e94c admin admin] REQUEST_METHOD: `PUT` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:30.059795 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c859edd-46f0-4734-b37a-71931430e94c admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:30.060061 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c859edd-46f0-4734-b37a-71931430e94c admin admin] PATH_INFO: `/v3/projects/7d2e3b7326644d01ab8000f8575c479b/users/ad5a8a3bea3a44019965d8c93730003c/roles/9d9d5d2c270146488249b7ed943280db` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:30.061907 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c859edd-46f0-4734-b37a-71931430e94c admin admin] RBAC: Authorizing `identity:create_grant(project_id=7d2e3b7326644d01ab8000f8575c479b, user_id=ad5a8a3bea3a44019965d8c93730003c, role_id=9d9d5d2c270146488249b7ed943280db)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:30.064584 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c859edd-46f0-4734-b37a-71931430e94c admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:30.074490 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1101/2230] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:13:30 2022] PUT /identity/v3/projects/7d2e3b7326644d01ab8000f8575c479b/users/ad5a8a3bea3a44019965d8c93730003c/roles/9d9d5d2c270146488249b7ed943280db => generated 0 bytes in 31 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:13:30.085085 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1711322-afc0-406e-958a-39dfd4416366 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:30.085370 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1711322-afc0-406e-958a-39dfd4416366 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:30.085638 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1711322-afc0-406e-958a-39dfd4416366 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:30.120225 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-e1711322-afc0-406e-958a-39dfd4416366 None None] MFA Rules not processed for user `ad5a8a3bea3a44019965d8c93730003c`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:13:30.121455 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-e1711322-afc0-406e-958a-39dfd4416366 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:30.190985 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1130/2231] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:13:30 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:13:30.202259 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c859edd-46f0-4734-b37a-71931430e94c admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:30.204008 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-2c859edd-46f0-4734-b37a-71931430e94c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:30.263408 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c859edd-46f0-4734-b37a-71931430e94c admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:30.267175 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23e2621d-a07c-4012-a2d2-e3d893896e63 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '019a79b2affe478b8a30d1c538d0415c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:30.268408 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23e2621d-a07c-4012-a2d2-e3d893896e63 service cinder] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:30.268665 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23e2621d-a07c-4012-a2d2-e3d893896e63 service cinder] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:30.268832 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23e2621d-a07c-4012-a2d2-e3d893896e63 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:30.279177 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23e2621d-a07c-4012-a2d2-e3d893896e63 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:30.280603 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23e2621d-a07c-4012-a2d2-e3d893896e63 service cinder] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:30.290949 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1102/2232] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:13:30 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:13:30.565066 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ab3c7e9-c7d8-4dea-8aca-fece83a5f93c None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:30.565552 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ab3c7e9-c7d8-4dea-8aca-fece83a5f93c None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:30.565645 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ab3c7e9-c7d8-4dea-8aca-fece83a5f93c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:30.589744 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-0ab3c7e9-c7d8-4dea-8aca-fece83a5f93c None None] MFA Rules not processed for user `71e9c605498544c8b6c75349d9b66a94`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:13:30.591006 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-0ab3c7e9-c7d8-4dea-8aca-fece83a5f93c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:30.665518 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1131/2233] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:13:30 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:13:30.882893 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23e2621d-a07c-4012-a2d2-e3d893896e63 service cinder] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:30.956767 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23e2621d-a07c-4012-a2d2-e3d893896e63 service cinder] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:30.961492 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf48ca44-af7d-4ea7-b6b2-5896ee25c347 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:30.962126 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf48ca44-af7d-4ea7-b6b2-5896ee25c347 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:30.962413 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf48ca44-af7d-4ea7-b6b2-5896ee25c347 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:30.962690 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf48ca44-af7d-4ea7-b6b2-5896ee25c347 admin admin] PATH_INFO: `/v3/users/08f2705fc76b4ae192626fb4de9ae6a3` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:30.963853 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf48ca44-af7d-4ea7-b6b2-5896ee25c347 admin admin] RBAC: Authorizing `identity:delete_user(user_id=08f2705fc76b4ae192626fb4de9ae6a3)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:30.965774 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf48ca44-af7d-4ea7-b6b2-5896ee25c347 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:31.016792 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=90,keystone:INSERT=5,keystone:DELETE=5 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:31.037191 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-bf48ca44-af7d-4ea7-b6b2-5896ee25c347 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '08f2705fc76b4ae192626fb4de9ae6a3'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:31.037883 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-bf48ca44-af7d-4ea7-b6b2-5896ee25c347 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '08f2705fc76b4ae192626fb4de9ae6a3'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:31.038493 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-bf48ca44-af7d-4ea7-b6b2-5896ee25c347 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '08f2705fc76b4ae192626fb4de9ae6a3'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:31.057745 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-bf48ca44-af7d-4ea7-b6b2-5896ee25c347 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '08f2705fc76b4ae192626fb4de9ae6a3'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:31.066001 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-bf48ca44-af7d-4ea7-b6b2-5896ee25c347 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '08f2705fc76b4ae192626fb4de9ae6a3'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:31.080523 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1103/2234] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:13:30 2022] DELETE /identity/v3/users/08f2705fc76b4ae192626fb4de9ae6a3 => generated 0 bytes in 199 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:13:31.085104 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa6b2bb-ff5b-4f27-b56c-a56c9a5ae8e7 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:31.085449 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa6b2bb-ff5b-4f27-b56c-a56c9a5ae8e7 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:31.085749 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa6b2bb-ff5b-4f27-b56c-a56c9a5ae8e7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:31.118592 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-1fa6b2bb-ff5b-4f27-b56c-a56c9a5ae8e7 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:13:31.119907 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-1fa6b2bb-ff5b-4f27-b56c-a56c9a5ae8e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:31.196834 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1132/2235] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:13:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:13:31.202765 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf48ca44-af7d-4ea7-b6b2-5896ee25c347 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:31.204188 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-bf48ca44-af7d-4ea7-b6b2-5896ee25c347 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:31.286658 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf48ca44-af7d-4ea7-b6b2-5896ee25c347 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:31.286658 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29b39476-0047-40ff-821d-292d3454397e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:31.286658 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29b39476-0047-40ff-821d-292d3454397e service neutron] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:31.286658 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29b39476-0047-40ff-821d-292d3454397e service neutron] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:31.287492 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29b39476-0047-40ff-821d-292d3454397e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:31.294340 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29b39476-0047-40ff-821d-292d3454397e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:31.296041 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29b39476-0047-40ff-821d-292d3454397e service neutron] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:31.308530 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1104/2236] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:13:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:13:31.630979 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fa6b2bb-ff5b-4f27-b56c-a56c9a5ae8e7 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:31.633423 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-1fa6b2bb-ff5b-4f27-b56c-a56c9a5ae8e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:31.651290 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fa6b2bb-ff5b-4f27-b56c-a56c9a5ae8e7 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:31.657087 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf921f1d-41f4-4885-a826-be6db6ad1a2f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:31.657723 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf921f1d-41f4-4885-a826-be6db6ad1a2f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:31.658005 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf921f1d-41f4-4885-a826-be6db6ad1a2f admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:31.658844 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf921f1d-41f4-4885-a826-be6db6ad1a2f admin admin] PATH_INFO: `/v3/projects/8a84d854a2164b2e880984e5f277d574` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:31.660172 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf921f1d-41f4-4885-a826-be6db6ad1a2f admin admin] RBAC: Authorizing `identity:delete_project(project_id=8a84d854a2164b2e880984e5f277d574)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:31.662009 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf921f1d-41f4-4885-a826-be6db6ad1a2f admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:31.694057 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=99,keystone:DELETE=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:31.713874 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-bf921f1d-41f4-4885-a826-be6db6ad1a2f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8a84d854a2164b2e880984e5f277d574', 'request_id': 'req-bf921f1d-41f4-4885-a826-be6db6ad1a2f'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:31.722388 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-bf921f1d-41f4-4885-a826-be6db6ad1a2f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8a84d854a2164b2e880984e5f277d574', 'request_id': 'req-bf921f1d-41f4-4885-a826-be6db6ad1a2f'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:31.730376 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-bf921f1d-41f4-4885-a826-be6db6ad1a2f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8a84d854a2164b2e880984e5f277d574', 'request_id': 'req-bf921f1d-41f4-4885-a826-be6db6ad1a2f'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:31.744175 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-bf921f1d-41f4-4885-a826-be6db6ad1a2f admin admin] The token cache is being invalidate because project 8a84d854a2164b2e880984e5f277d574 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81625) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:13:31.744652 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-bf921f1d-41f4-4885-a826-be6db6ad1a2f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:31.745506 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-bf921f1d-41f4-4885-a826-be6db6ad1a2f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:31.747191 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1133/2237] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:13:31 2022] DELETE /identity/v3/projects/8a84d854a2164b2e880984e5f277d574 => generated 0 bytes in 117 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:13:31.751438 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29b39476-0047-40ff-821d-292d3454397e service neutron] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:31.761351 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29b39476-0047-40ff-821d-292d3454397e service neutron] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:31.827850 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40a39f51-117c-4032-b043-fdff28a37ae9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:31.828472 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a39f51-117c-4032-b043-fdff28a37ae9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:31.828744 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a39f51-117c-4032-b043-fdff28a37ae9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:31.829010 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a39f51-117c-4032-b043-fdff28a37ae9 admin admin] PATH_INFO: `/v3/users/ad5a8a3bea3a44019965d8c93730003c` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:31.830636 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40a39f51-117c-4032-b043-fdff28a37ae9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ad5a8a3bea3a44019965d8c93730003c)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:31.832578 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40a39f51-117c-4032-b043-fdff28a37ae9 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:31.880584 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-40a39f51-117c-4032-b043-fdff28a37ae9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ad5a8a3bea3a44019965d8c93730003c'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:31.881239 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-40a39f51-117c-4032-b043-fdff28a37ae9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ad5a8a3bea3a44019965d8c93730003c'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:31.881841 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-40a39f51-117c-4032-b043-fdff28a37ae9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ad5a8a3bea3a44019965d8c93730003c'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:31.887188 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:31.895716 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-40a39f51-117c-4032-b043-fdff28a37ae9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ad5a8a3bea3a44019965d8c93730003c'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:31.902988 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-40a39f51-117c-4032-b043-fdff28a37ae9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ad5a8a3bea3a44019965d8c93730003c'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:31.916817 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1105/2238] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:13:31 2022] DELETE /identity/v3/users/ad5a8a3bea3a44019965d8c93730003c => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 05:13:32.224060 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf921f1d-41f4-4885-a826-be6db6ad1a2f admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:32.225538 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-bf921f1d-41f4-4885-a826-be6db6ad1a2f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:32.242280 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf921f1d-41f4-4885-a826-be6db6ad1a2f admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:32.300359 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e49b6aad-78ca-4185-9215-cf6c9d3248d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:32.300982 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49b6aad-78ca-4185-9215-cf6c9d3248d4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:32.301256 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49b6aad-78ca-4185-9215-cf6c9d3248d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:32.301525 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49b6aad-78ca-4185-9215-cf6c9d3248d4 admin admin] PATH_INFO: `/v3/projects/7d2e3b7326644d01ab8000f8575c479b` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:32.302613 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e49b6aad-78ca-4185-9215-cf6c9d3248d4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7d2e3b7326644d01ab8000f8575c479b)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:32.304543 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e49b6aad-78ca-4185-9215-cf6c9d3248d4 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:32.351077 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-e49b6aad-78ca-4185-9215-cf6c9d3248d4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7d2e3b7326644d01ab8000f8575c479b', 'request_id': 'req-e49b6aad-78ca-4185-9215-cf6c9d3248d4'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:32.359184 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-e49b6aad-78ca-4185-9215-cf6c9d3248d4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7d2e3b7326644d01ab8000f8575c479b', 'request_id': 'req-e49b6aad-78ca-4185-9215-cf6c9d3248d4'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:32.366218 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-e49b6aad-78ca-4185-9215-cf6c9d3248d4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7d2e3b7326644d01ab8000f8575c479b', 'request_id': 'req-e49b6aad-78ca-4185-9215-cf6c9d3248d4'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:32.383072 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-e49b6aad-78ca-4185-9215-cf6c9d3248d4 admin admin] The token cache is being invalidate because project 7d2e3b7326644d01ab8000f8575c479b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81625) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:13:32.383072 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-e49b6aad-78ca-4185-9215-cf6c9d3248d4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:32.383072 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-e49b6aad-78ca-4185-9215-cf6c9d3248d4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:32.383072 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1134/2239] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:13:32 2022] DELETE /identity/v3/projects/7d2e3b7326644d01ab8000f8575c479b => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:13:32.385173 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d7c95f-31e1-4658-828f-9ef09e1c3a64 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:32.385639 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d7c95f-31e1-4658-828f-9ef09e1c3a64 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:32.385971 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d7c95f-31e1-4658-828f-9ef09e1c3a64 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:32.405219 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-61d7c95f-31e1-4658-828f-9ef09e1c3a64 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:13:32.406493 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-61d7c95f-31e1-4658-828f-9ef09e1c3a64 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:32.467416 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1106/2240] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:13:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:13:32.477593 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99c2daca-440b-4f41-a4ea-8ad6b85225ac None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:32.477888 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99c2daca-440b-4f41-a4ea-8ad6b85225ac None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:32.478161 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99c2daca-440b-4f41-a4ea-8ad6b85225ac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:32.498741 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-99c2daca-440b-4f41-a4ea-8ad6b85225ac None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:13:32.499920 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-99c2daca-440b-4f41-a4ea-8ad6b85225ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:32.508777 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1135/2241] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:13:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:13:32.513101 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61d7c95f-31e1-4658-828f-9ef09e1c3a64 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:32.522893 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61d7c95f-31e1-4658-828f-9ef09e1c3a64 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:32.527405 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82cb0d42-13d3-476a-b19f-4246574bd097 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:32.527999 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82cb0d42-13d3-476a-b19f-4246574bd097 admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:32.528275 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82cb0d42-13d3-476a-b19f-4246574bd097 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:32.528547 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82cb0d42-13d3-476a-b19f-4246574bd097 admin admin] PATH_INFO: `/v3/domains` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:32.529189 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82cb0d42-13d3-476a-b19f-4246574bd097 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:13:32.529549 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82cb0d42-13d3-476a-b19f-4246574bd097 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:32.531192 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82cb0d42-13d3-476a-b19f-4246574bd097 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:32.542787 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1107/2242] 10.222.0.77 () {58 vars in 1153 bytes} [Tue Aug 30 05:13:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:13:32.547267 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99c2daca-440b-4f41-a4ea-8ad6b85225ac None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:32.557149 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99c2daca-440b-4f41-a4ea-8ad6b85225ac None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:32.562130 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a898fc32-b7a4-40a3-b119-44b9071b81d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:32.562731 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a898fc32-b7a4-40a3-b119-44b9071b81d1 admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:32.563045 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a898fc32-b7a4-40a3-b119-44b9071b81d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:32.563319 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a898fc32-b7a4-40a3-b119-44b9071b81d1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:32.564024 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a898fc32-b7a4-40a3-b119-44b9071b81d1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:32.566399 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a898fc32-b7a4-40a3-b119-44b9071b81d1 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:32.596189 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1136/2243] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:13:32 2022] POST /identity/v3/projects => generated 387 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:13:32.600307 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82cb0d42-13d3-476a-b19f-4246574bd097 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:32.610227 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82cb0d42-13d3-476a-b19f-4246574bd097 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:32.668819 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-288765a3-5f23-4ae5-8ad0-479848e012ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:32.669431 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-288765a3-5f23-4ae5-8ad0-479848e012ad admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:32.669742 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-288765a3-5f23-4ae5-8ad0-479848e012ad admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:32.670024 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-288765a3-5f23-4ae5-8ad0-479848e012ad admin admin] PATH_INFO: `/v3/users` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:32.670664 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-288765a3-5f23-4ae5-8ad0-479848e012ad admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:32.672664 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-288765a3-5f23-4ae5-8ad0-479848e012ad admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:32.694500 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1108/2244] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:13:32 2022] POST /identity/v3/users => generated 294 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 05:13:32.698554 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a898fc32-b7a4-40a3-b119-44b9071b81d1 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:32.709084 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a898fc32-b7a4-40a3-b119-44b9071b81d1 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:32.713905 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e173da9-f914-48aa-b359-0be33592d00b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:32.714489 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e173da9-f914-48aa-b359-0be33592d00b admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:32.714764 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e173da9-f914-48aa-b359-0be33592d00b admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:32.715062 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e173da9-f914-48aa-b359-0be33592d00b admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:32.715695 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e173da9-f914-48aa-b359-0be33592d00b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:32.717258 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e173da9-f914-48aa-b359-0be33592d00b admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:32.727757 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1137/2245] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:13:32 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:13:32.731486 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-288765a3-5f23-4ae5-8ad0-479848e012ad admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:32.742099 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-288765a3-5f23-4ae5-8ad0-479848e012ad admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:32.746697 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2d2611d-b608-4501-9fa8-c4aa97ebe5e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:32.747321 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2d2611d-b608-4501-9fa8-c4aa97ebe5e4 admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:32.747602 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2d2611d-b608-4501-9fa8-c4aa97ebe5e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:32.747871 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2d2611d-b608-4501-9fa8-c4aa97ebe5e4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:32.748498 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2d2611d-b608-4501-9fa8-c4aa97ebe5e4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:32.750123 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2d2611d-b608-4501-9fa8-c4aa97ebe5e4 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:32.760966 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1109/2246] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:13:32 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:13:32.765240 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e173da9-f914-48aa-b359-0be33592d00b admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:32.771498 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:DELETE=13,keystone:SELECT=84,keystone:INSERT=3 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:32.775516 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e173da9-f914-48aa-b359-0be33592d00b admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:32.781269 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f63dc371-08ee-4622-bfd7-090bbc219906 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:32.782415 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f63dc371-08ee-4622-bfd7-090bbc219906 admin admin] REQUEST_METHOD: `PUT` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:32.782980 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f63dc371-08ee-4622-bfd7-090bbc219906 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:32.783889 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f63dc371-08ee-4622-bfd7-090bbc219906 admin admin] PATH_INFO: `/v3/projects/6785bf3addad4231823525dc5a221696/users/85b3213407fa498ab5a26b59c11fc4c2/roles/6f03d86980c34c36b44a8d6764ff4fda` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:32.799821 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f63dc371-08ee-4622-bfd7-090bbc219906 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6785bf3addad4231823525dc5a221696, user_id=85b3213407fa498ab5a26b59c11fc4c2, role_id=6f03d86980c34c36b44a8d6764ff4fda)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:32.802460 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f63dc371-08ee-4622-bfd7-090bbc219906 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:32.814969 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1138/2247] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:13:32 2022] PUT /identity/v3/projects/6785bf3addad4231823525dc5a221696/users/85b3213407fa498ab5a26b59c11fc4c2/roles/6f03d86980c34c36b44a8d6764ff4fda => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 05:13:32.825200 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da7a5cf1-599e-4cf6-a279-a407bdbd7601 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:32.825489 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da7a5cf1-599e-4cf6-a279-a407bdbd7601 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:32.825759 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da7a5cf1-599e-4cf6-a279-a407bdbd7601 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:32.853985 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-da7a5cf1-599e-4cf6-a279-a407bdbd7601 None None] MFA Rules not processed for user `85b3213407fa498ab5a26b59c11fc4c2`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:13:32.855220 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-da7a5cf1-599e-4cf6-a279-a407bdbd7601 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:32.880949 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=94,keystone:DELETE=2,keystone:INSERT=4 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:32.919676 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1110/2248] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:13:32 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:13:32.924876 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f63dc371-08ee-4622-bfd7-090bbc219906 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:32.985501 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f63dc371-08ee-4622-bfd7-090bbc219906 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:32.990672 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d133413-4f10-4d3e-b8a5-ff3e064822fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:32.991300 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d133413-4f10-4d3e-b8a5-ff3e064822fe admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:32.991575 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d133413-4f10-4d3e-b8a5-ff3e064822fe admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:32.991838 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d133413-4f10-4d3e-b8a5-ff3e064822fe admin admin] PATH_INFO: `/v3/projects` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:32.992466 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d133413-4f10-4d3e-b8a5-ff3e064822fe admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:32.994339 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d133413-4f10-4d3e-b8a5-ff3e064822fe admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:33.019811 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1139/2249] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:13:32 2022] POST /identity/v3/projects => generated 383 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:13:33.023713 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da7a5cf1-599e-4cf6-a279-a407bdbd7601 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:33.035019 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da7a5cf1-599e-4cf6-a279-a407bdbd7601 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:33.090124 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ec95103-050a-430f-84c7-9a7ab3282599 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:33.090724 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ec95103-050a-430f-84c7-9a7ab3282599 admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:33.091039 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ec95103-050a-430f-84c7-9a7ab3282599 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:33.091316 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ec95103-050a-430f-84c7-9a7ab3282599 admin admin] PATH_INFO: `/v3/users` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:33.091947 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ec95103-050a-430f-84c7-9a7ab3282599 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:33.094140 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ec95103-050a-430f-84c7-9a7ab3282599 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:33.117227 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1111/2250] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:13:33 2022] POST /identity/v3/users => generated 298 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:13:33.120983 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d133413-4f10-4d3e-b8a5-ff3e064822fe admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:33.131377 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d133413-4f10-4d3e-b8a5-ff3e064822fe admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:33.136278 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3af53f28-9187-4d4f-90d9-d5a84953657a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:33.136871 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3af53f28-9187-4d4f-90d9-d5a84953657a admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:33.137150 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3af53f28-9187-4d4f-90d9-d5a84953657a admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:33.137426 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3af53f28-9187-4d4f-90d9-d5a84953657a admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:33.138066 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3af53f28-9187-4d4f-90d9-d5a84953657a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:33.140018 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3af53f28-9187-4d4f-90d9-d5a84953657a admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:33.151984 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1140/2251] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:13:33 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 05:13:33.155996 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ec95103-050a-430f-84c7-9a7ab3282599 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:33.166614 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ec95103-050a-430f-84c7-9a7ab3282599 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:33.171330 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b896dd9e-8fce-4cbb-838c-69b2159bcd38 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:33.171971 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b896dd9e-8fce-4cbb-838c-69b2159bcd38 admin admin] REQUEST_METHOD: `PUT` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:33.172252 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b896dd9e-8fce-4cbb-838c-69b2159bcd38 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:33.172524 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b896dd9e-8fce-4cbb-838c-69b2159bcd38 admin admin] PATH_INFO: `/v3/projects/c29ce438f1bc43c49cb4e4bfe11c6133/users/2c52ba37640549ed858df73dfa60ef57/roles/9d9d5d2c270146488249b7ed943280db` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:33.187242 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b896dd9e-8fce-4cbb-838c-69b2159bcd38 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c29ce438f1bc43c49cb4e4bfe11c6133, user_id=2c52ba37640549ed858df73dfa60ef57, role_id=9d9d5d2c270146488249b7ed943280db)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:33.189880 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b896dd9e-8fce-4cbb-838c-69b2159bcd38 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:33.203195 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1112/2252] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:13:33 2022] PUT /identity/v3/projects/c29ce438f1bc43c49cb4e4bfe11c6133/users/2c52ba37640549ed858df73dfa60ef57/roles/9d9d5d2c270146488249b7ed943280db => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:13:33.207609 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3af53f28-9187-4d4f-90d9-d5a84953657a admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:33.217697 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3af53f28-9187-4d4f-90d9-d5a84953657a admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:33.275850 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a55bf85-5354-4485-b06d-c434a3b78a4d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:33.276367 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a55bf85-5354-4485-b06d-c434a3b78a4d admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:33.276640 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a55bf85-5354-4485-b06d-c434a3b78a4d admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:33.276903 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a55bf85-5354-4485-b06d-c434a3b78a4d admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:33.277523 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a55bf85-5354-4485-b06d-c434a3b78a4d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:33.279119 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a55bf85-5354-4485-b06d-c434a3b78a4d admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:33.289437 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1141/2253] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:13:33 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:13:33.293537 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b896dd9e-8fce-4cbb-838c-69b2159bcd38 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:33.303526 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b896dd9e-8fce-4cbb-838c-69b2159bcd38 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:33.308089 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2efac073-343d-4ec6-bb7e-920ec2f06f6d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:33.308716 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2efac073-343d-4ec6-bb7e-920ec2f06f6d admin admin] REQUEST_METHOD: `PUT` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:33.308991 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2efac073-343d-4ec6-bb7e-920ec2f06f6d admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:33.309257 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2efac073-343d-4ec6-bb7e-920ec2f06f6d admin admin] PATH_INFO: `/v3/projects/c29ce438f1bc43c49cb4e4bfe11c6133/users/2c52ba37640549ed858df73dfa60ef57/roles/9d9d5d2c270146488249b7ed943280db` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:33.311166 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2efac073-343d-4ec6-bb7e-920ec2f06f6d admin admin] RBAC: Authorizing `identity:create_grant(project_id=c29ce438f1bc43c49cb4e4bfe11c6133, user_id=2c52ba37640549ed858df73dfa60ef57, role_id=9d9d5d2c270146488249b7ed943280db)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:33.315771 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2efac073-343d-4ec6-bb7e-920ec2f06f6d admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:33.326487 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1113/2254] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:13:33 2022] PUT /identity/v3/projects/c29ce438f1bc43c49cb4e4bfe11c6133/users/2c52ba37640549ed858df73dfa60ef57/roles/9d9d5d2c270146488249b7ed943280db => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:13:33.337658 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e854dd-7295-4bcd-89f3-71fcfa6991da None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:33.337943 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e854dd-7295-4bcd-89f3-71fcfa6991da None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:33.338214 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e854dd-7295-4bcd-89f3-71fcfa6991da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:33.365672 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-e7e854dd-7295-4bcd-89f3-71fcfa6991da None None] MFA Rules not processed for user `2c52ba37640549ed858df73dfa60ef57`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:13:33.366883 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-e7e854dd-7295-4bcd-89f3-71fcfa6991da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:33.406690 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:33.437350 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1142/2255] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:13:33 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:13:33.448994 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2efac073-343d-4ec6-bb7e-920ec2f06f6d admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:33.450774 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-2efac073-343d-4ec6-bb7e-920ec2f06f6d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:33.510405 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2efac073-343d-4ec6-bb7e-920ec2f06f6d admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:33.514041 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f325b72-d5c9-4865-b745-a3bf57011a11 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '019a79b2affe478b8a30d1c538d0415c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:33.514641 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f325b72-d5c9-4865-b745-a3bf57011a11 service cinder] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:33.514962 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f325b72-d5c9-4865-b745-a3bf57011a11 service cinder] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:33.515236 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f325b72-d5c9-4865-b745-a3bf57011a11 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:33.524966 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f325b72-d5c9-4865-b745-a3bf57011a11 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:33.526473 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f325b72-d5c9-4865-b745-a3bf57011a11 service cinder] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:33.535841 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1114/2256] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:13:33 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:13:33.610719 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7e854dd-7295-4bcd-89f3-71fcfa6991da None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:33.675528 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7e854dd-7295-4bcd-89f3-71fcfa6991da None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:33.680126 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-845447b3-ed5c-4d76-b727-84f94c3e741e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:33.681816 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-845447b3-ed5c-4d76-b727-84f94c3e741e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:33.681816 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-845447b3-ed5c-4d76-b727-84f94c3e741e admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:33.681816 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-845447b3-ed5c-4d76-b727-84f94c3e741e admin admin] PATH_INFO: `/v3/users/85b3213407fa498ab5a26b59c11fc4c2` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:33.682379 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-845447b3-ed5c-4d76-b727-84f94c3e741e admin admin] RBAC: Authorizing `identity:delete_user(user_id=85b3213407fa498ab5a26b59c11fc4c2)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:33.684430 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-845447b3-ed5c-4d76-b727-84f94c3e741e admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:33.762393 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-845447b3-ed5c-4d76-b727-84f94c3e741e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '85b3213407fa498ab5a26b59c11fc4c2'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:33.785926 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-845447b3-ed5c-4d76-b727-84f94c3e741e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '85b3213407fa498ab5a26b59c11fc4c2'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:33.801296 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-845447b3-ed5c-4d76-b727-84f94c3e741e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '85b3213407fa498ab5a26b59c11fc4c2'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:33.810385 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-845447b3-ed5c-4d76-b727-84f94c3e741e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '85b3213407fa498ab5a26b59c11fc4c2'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:33.811376 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-845447b3-ed5c-4d76-b727-84f94c3e741e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '85b3213407fa498ab5a26b59c11fc4c2'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:33.818797 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1143/2257] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:13:33 2022] DELETE /identity/v3/users/85b3213407fa498ab5a26b59c11fc4c2 => generated 0 bytes in 208 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:13:33.824157 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-062ce1b3-1521-4c58-b12d-fc4757019e6a None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:33.824452 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-062ce1b3-1521-4c58-b12d-fc4757019e6a None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:33.824956 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-062ce1b3-1521-4c58-b12d-fc4757019e6a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:33.855995 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-062ce1b3-1521-4c58-b12d-fc4757019e6a None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:13:33.856985 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-062ce1b3-1521-4c58-b12d-fc4757019e6a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:33.878302 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:33.930467 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1115/2258] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:13:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:13:33.943207 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-845447b3-ed5c-4d76-b727-84f94c3e741e admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:33.944793 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-845447b3-ed5c-4d76-b727-84f94c3e741e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:34.012725 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-845447b3-ed5c-4d76-b727-84f94c3e741e admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:34.016508 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b76a1f5d-94bc-45c7-8083-d8b6a73cd5f6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:34.017665 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b76a1f5d-94bc-45c7-8083-d8b6a73cd5f6 service neutron] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:34.017951 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b76a1f5d-94bc-45c7-8083-d8b6a73cd5f6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:34.018222 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b76a1f5d-94bc-45c7-8083-d8b6a73cd5f6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:34.030232 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b76a1f5d-94bc-45c7-8083-d8b6a73cd5f6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:34.032334 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b76a1f5d-94bc-45c7-8083-d8b6a73cd5f6 service neutron] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:34.037823 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:34.055048 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1144/2259] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:13:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:13:34.317880 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-062ce1b3-1521-4c58-b12d-fc4757019e6a None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:34.319395 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-062ce1b3-1521-4c58-b12d-fc4757019e6a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:34.331717 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-062ce1b3-1521-4c58-b12d-fc4757019e6a None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:34.337238 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b79a160-da6f-43fb-83cd-02aa8fa94ead admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:34.338700 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b79a160-da6f-43fb-83cd-02aa8fa94ead admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:34.338930 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b79a160-da6f-43fb-83cd-02aa8fa94ead admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:34.339246 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b79a160-da6f-43fb-83cd-02aa8fa94ead admin admin] PATH_INFO: `/v3/projects/6785bf3addad4231823525dc5a221696` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:34.340370 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b79a160-da6f-43fb-83cd-02aa8fa94ead admin admin] RBAC: Authorizing `identity:delete_project(project_id=6785bf3addad4231823525dc5a221696)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:34.342282 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b79a160-da6f-43fb-83cd-02aa8fa94ead admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:34.397952 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-3b79a160-da6f-43fb-83cd-02aa8fa94ead admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6785bf3addad4231823525dc5a221696', 'request_id': 'req-3b79a160-da6f-43fb-83cd-02aa8fa94ead'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:34.409716 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-3b79a160-da6f-43fb-83cd-02aa8fa94ead admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6785bf3addad4231823525dc5a221696', 'request_id': 'req-3b79a160-da6f-43fb-83cd-02aa8fa94ead'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:34.418429 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-3b79a160-da6f-43fb-83cd-02aa8fa94ead admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6785bf3addad4231823525dc5a221696', 'request_id': 'req-3b79a160-da6f-43fb-83cd-02aa8fa94ead'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:34.441244 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-3b79a160-da6f-43fb-83cd-02aa8fa94ead admin admin] The token cache is being invalidate because project 6785bf3addad4231823525dc5a221696 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81624) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:13:34.441602 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-3b79a160-da6f-43fb-83cd-02aa8fa94ead admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:34.442306 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-3b79a160-da6f-43fb-83cd-02aa8fa94ead admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:34.443954 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1116/2260] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:13:34 2022] DELETE /identity/v3/projects/6785bf3addad4231823525dc5a221696 => generated 0 bytes in 126 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:13:34.448506 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b76a1f5d-94bc-45c7-8083-d8b6a73cd5f6 service neutron] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:34.464415 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b76a1f5d-94bc-45c7-8083-d8b6a73cd5f6 service neutron] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:34.548113 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8c251c6-a190-46ad-a636-1db8062ce7e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:34.549878 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8c251c6-a190-46ad-a636-1db8062ce7e4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:34.550165 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8c251c6-a190-46ad-a636-1db8062ce7e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:34.550435 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8c251c6-a190-46ad-a636-1db8062ce7e4 admin admin] PATH_INFO: `/v3/users/2c52ba37640549ed858df73dfa60ef57` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:34.551686 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8c251c6-a190-46ad-a636-1db8062ce7e4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2c52ba37640549ed858df73dfa60ef57)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:34.553917 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8c251c6-a190-46ad-a636-1db8062ce7e4 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:34.605575 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-c8c251c6-a190-46ad-a636-1db8062ce7e4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2c52ba37640549ed858df73dfa60ef57'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:34.612322 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-c8c251c6-a190-46ad-a636-1db8062ce7e4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2c52ba37640549ed858df73dfa60ef57'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:34.624728 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-c8c251c6-a190-46ad-a636-1db8062ce7e4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2c52ba37640549ed858df73dfa60ef57'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:34.631472 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-c8c251c6-a190-46ad-a636-1db8062ce7e4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2c52ba37640549ed858df73dfa60ef57'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:34.632175 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-c8c251c6-a190-46ad-a636-1db8062ce7e4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2c52ba37640549ed858df73dfa60ef57'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:34.638099 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1145/2261] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:13:34 2022] DELETE /identity/v3/users/2c52ba37640549ed858df73dfa60ef57 => generated 0 bytes in 191 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 05:13:34.891360 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5022b18-b4d4-4e1f-8b7f-73f00810489d None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:34.892199 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5022b18-b4d4-4e1f-8b7f-73f00810489d None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:34.892992 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5022b18-b4d4-4e1f-8b7f-73f00810489d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:34.918228 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-d5022b18-b4d4-4e1f-8b7f-73f00810489d None None] MFA Rules not processed for user `b50d70f00f33494e81d4aac49df96dc4`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:13:34.920126 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8c251c6-a190-46ad-a636-1db8062ce7e4 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:34.921067 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-d5022b18-b4d4-4e1f-8b7f-73f00810489d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:34.921797 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-c8c251c6-a190-46ad-a636-1db8062ce7e4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:35.005578 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1117/2262] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:13:34 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:13:35.028799 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8c251c6-a190-46ad-a636-1db8062ce7e4 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:35.036665 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce1e1ebc-6b35-4716-9a23-7b98d8044048 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:35.037059 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce1e1ebc-6b35-4716-9a23-7b98d8044048 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:35.037341 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce1e1ebc-6b35-4716-9a23-7b98d8044048 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:35.037661 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce1e1ebc-6b35-4716-9a23-7b98d8044048 admin admin] PATH_INFO: `/v3/projects/c29ce438f1bc43c49cb4e4bfe11c6133` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:35.038830 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce1e1ebc-6b35-4716-9a23-7b98d8044048 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c29ce438f1bc43c49cb4e4bfe11c6133)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:35.041085 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce1e1ebc-6b35-4716-9a23-7b98d8044048 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:35.068623 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:35.103686 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-ce1e1ebc-6b35-4716-9a23-7b98d8044048 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c29ce438f1bc43c49cb4e4bfe11c6133', 'request_id': 'req-ce1e1ebc-6b35-4716-9a23-7b98d8044048'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:35.123090 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-ce1e1ebc-6b35-4716-9a23-7b98d8044048 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c29ce438f1bc43c49cb4e4bfe11c6133', 'request_id': 'req-ce1e1ebc-6b35-4716-9a23-7b98d8044048'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:35.131282 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-ce1e1ebc-6b35-4716-9a23-7b98d8044048 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c29ce438f1bc43c49cb4e4bfe11c6133', 'request_id': 'req-ce1e1ebc-6b35-4716-9a23-7b98d8044048'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:35.147979 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-ce1e1ebc-6b35-4716-9a23-7b98d8044048 admin admin] The token cache is being invalidate because project c29ce438f1bc43c49cb4e4bfe11c6133 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81625) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:13:35.148275 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-ce1e1ebc-6b35-4716-9a23-7b98d8044048 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:35.148940 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-ce1e1ebc-6b35-4716-9a23-7b98d8044048 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:35.150650 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1146/2263] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:13:34 2022] DELETE /identity/v3/projects/c29ce438f1bc43c49cb4e4bfe11c6133 => generated 0 bytes in 231 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 05:13:35.155391 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6ff1fdd-d5f0-4a06-9ba7-51af4e200059 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:35.155679 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6ff1fdd-d5f0-4a06-9ba7-51af4e200059 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:35.155951 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6ff1fdd-d5f0-4a06-9ba7-51af4e200059 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:35.179120 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-c6ff1fdd-d5f0-4a06-9ba7-51af4e200059 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:13:35.180331 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-c6ff1fdd-d5f0-4a06-9ba7-51af4e200059 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:35.238952 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:35.262538 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1118/2264] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:13:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 05:13:35.270375 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b948a23-3af6-4941-9a09-1f4f567cc10a None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:35.270660 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b948a23-3af6-4941-9a09-1f4f567cc10a None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:35.270969 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b948a23-3af6-4941-9a09-1f4f567cc10a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:35.291767 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-0b948a23-3af6-4941-9a09-1f4f567cc10a None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:13:35.293866 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-0b948a23-3af6-4941-9a09-1f4f567cc10a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:35.303294 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1147/2265] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:13:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 05:13:35.307946 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6ff1fdd-d5f0-4a06-9ba7-51af4e200059 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:35.321512 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6ff1fdd-d5f0-4a06-9ba7-51af4e200059 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:35.327786 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d535e3b-c8a5-4662-9b87-1518ca902d4a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:35.327786 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d535e3b-c8a5-4662-9b87-1518ca902d4a admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:35.328246 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d535e3b-c8a5-4662-9b87-1518ca902d4a admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:35.328619 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d535e3b-c8a5-4662-9b87-1518ca902d4a admin admin] PATH_INFO: `/v3/domains` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:35.329379 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d535e3b-c8a5-4662-9b87-1518ca902d4a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:13:35.329783 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d535e3b-c8a5-4662-9b87-1518ca902d4a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:35.332081 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d535e3b-c8a5-4662-9b87-1518ca902d4a admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:35.348923 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1119/2266] 10.222.0.77 () {58 vars in 1153 bytes} [Tue Aug 30 05:13:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 05:13:35.352745 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b948a23-3af6-4941-9a09-1f4f567cc10a None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:35.362713 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b948a23-3af6-4941-9a09-1f4f567cc10a None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:35.375127 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-903c7713-e7b9-46a3-a29d-de91d8cc2765 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:35.375700 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-903c7713-e7b9-46a3-a29d-de91d8cc2765 admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:35.375977 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-903c7713-e7b9-46a3-a29d-de91d8cc2765 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:35.376240 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-903c7713-e7b9-46a3-a29d-de91d8cc2765 admin admin] PATH_INFO: `/v3/projects` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:35.377414 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-903c7713-e7b9-46a3-a29d-de91d8cc2765 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:35.379368 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-903c7713-e7b9-46a3-a29d-de91d8cc2765 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:35.412083 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1148/2267] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:13:35 2022] POST /identity/v3/projects => generated 393 bytes in 60 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:13:35.416556 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d535e3b-c8a5-4662-9b87-1518ca902d4a admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:35.430853 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d535e3b-c8a5-4662-9b87-1518ca902d4a admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:35.512597 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbe180c3-7678-4b67-8f1b-7c34bef550bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:35.513317 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbe180c3-7678-4b67-8f1b-7c34bef550bd admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:35.513700 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbe180c3-7678-4b67-8f1b-7c34bef550bd admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:35.514078 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbe180c3-7678-4b67-8f1b-7c34bef550bd admin admin] PATH_INFO: `/v3/users` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:35.514843 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbe180c3-7678-4b67-8f1b-7c34bef550bd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:35.516864 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbe180c3-7678-4b67-8f1b-7c34bef550bd admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:35.541489 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1120/2268] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:13:35 2022] POST /identity/v3/users => generated 297 bytes in 126 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:13:35.545622 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-903c7713-e7b9-46a3-a29d-de91d8cc2765 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:35.557587 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-903c7713-e7b9-46a3-a29d-de91d8cc2765 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:35.563740 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e99cb9c6-1a7b-40e6-874c-3d1868fc495e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:35.565083 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e99cb9c6-1a7b-40e6-874c-3d1868fc495e admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:35.565423 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e99cb9c6-1a7b-40e6-874c-3d1868fc495e admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:35.565760 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e99cb9c6-1a7b-40e6-874c-3d1868fc495e admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:35.566849 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e99cb9c6-1a7b-40e6-874c-3d1868fc495e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:35.568907 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e99cb9c6-1a7b-40e6-874c-3d1868fc495e admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:35.579530 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1149/2269] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:13:35 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:13:35.583907 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbe180c3-7678-4b67-8f1b-7c34bef550bd admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:35.595311 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbe180c3-7678-4b67-8f1b-7c34bef550bd admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:35.623426 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c08ada4f-0aa6-4cda-8014-3f534cafbf05 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:35.624042 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c08ada4f-0aa6-4cda-8014-3f534cafbf05 admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:35.624421 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c08ada4f-0aa6-4cda-8014-3f534cafbf05 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:35.624700 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c08ada4f-0aa6-4cda-8014-3f534cafbf05 admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:35.625353 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c08ada4f-0aa6-4cda-8014-3f534cafbf05 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:35.631862 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c08ada4f-0aa6-4cda-8014-3f534cafbf05 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:35.657437 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1121/2270] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:13:35 2022] GET /identity/v3/roles => generated 1420 bytes in 74 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 05:13:35.661499 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e99cb9c6-1a7b-40e6-874c-3d1868fc495e admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:35.676605 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e99cb9c6-1a7b-40e6-874c-3d1868fc495e admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:35.687678 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55b4e4d2-ec53-41bc-abfc-cfc7a927b3a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:35.688361 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b4e4d2-ec53-41bc-abfc-cfc7a927b3a7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:35.688670 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b4e4d2-ec53-41bc-abfc-cfc7a927b3a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:35.688942 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b4e4d2-ec53-41bc-abfc-cfc7a927b3a7 admin admin] PATH_INFO: `/v3/projects/e911a926e3314cdebb9633139261dcbd/users/272d9721fee94cb6af129c62809ac08d/roles/6f03d86980c34c36b44a8d6764ff4fda` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:35.712328 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55b4e4d2-ec53-41bc-abfc-cfc7a927b3a7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e911a926e3314cdebb9633139261dcbd, user_id=272d9721fee94cb6af129c62809ac08d, role_id=6f03d86980c34c36b44a8d6764ff4fda)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:35.715097 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55b4e4d2-ec53-41bc-abfc-cfc7a927b3a7 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:35.733807 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1150/2271] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:13:35 2022] PUT /identity/v3/projects/e911a926e3314cdebb9633139261dcbd/users/272d9721fee94cb6af129c62809ac08d/roles/6f03d86980c34c36b44a8d6764ff4fda => generated 0 bytes in 73 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:13:35.745132 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aaa45db-b99d-4d8a-a60e-0d6a59dec56e None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:35.745425 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aaa45db-b99d-4d8a-a60e-0d6a59dec56e None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:35.745699 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aaa45db-b99d-4d8a-a60e-0d6a59dec56e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:35.791382 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-6aaa45db-b99d-4d8a-a60e-0d6a59dec56e None None] MFA Rules not processed for user `272d9721fee94cb6af129c62809ac08d`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:13:35.792587 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-6aaa45db-b99d-4d8a-a60e-0d6a59dec56e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:35.860128 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1122/2272] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:13:35 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:13:35.866001 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55b4e4d2-ec53-41bc-abfc-cfc7a927b3a7 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:35.935451 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55b4e4d2-ec53-41bc-abfc-cfc7a927b3a7 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:35.940251 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c904aa8a-5f05-4cf0-bf97-5cd9b2724143 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:35.940852 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c904aa8a-5f05-4cf0-bf97-5cd9b2724143 admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:35.941130 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c904aa8a-5f05-4cf0-bf97-5cd9b2724143 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:35.941401 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c904aa8a-5f05-4cf0-bf97-5cd9b2724143 admin admin] PATH_INFO: `/v3/projects` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:35.942045 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c904aa8a-5f05-4cf0-bf97-5cd9b2724143 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:35.944002 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c904aa8a-5f05-4cf0-bf97-5cd9b2724143 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:35.972974 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1151/2273] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:13:35 2022] POST /identity/v3/projects => generated 393 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:13:35.977393 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aaa45db-b99d-4d8a-a60e-0d6a59dec56e None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:35.993325 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aaa45db-b99d-4d8a-a60e-0d6a59dec56e None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:36.043359 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:36.059755 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39d94abb-1310-4a34-8fb1-082a796a441c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:36.060662 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d94abb-1310-4a34-8fb1-082a796a441c admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:36.061018 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d94abb-1310-4a34-8fb1-082a796a441c admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:36.061365 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d94abb-1310-4a34-8fb1-082a796a441c admin admin] PATH_INFO: `/v3/users` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:36.062068 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39d94abb-1310-4a34-8fb1-082a796a441c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:36.064139 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39d94abb-1310-4a34-8fb1-082a796a441c admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:36.089377 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1123/2274] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:13:35 2022] POST /identity/v3/users => generated 303 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:13:36.090971 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c904aa8a-5f05-4cf0-bf97-5cd9b2724143 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:36.103131 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c904aa8a-5f05-4cf0-bf97-5cd9b2724143 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:36.107835 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad5ad9f4-7498-416b-9042-0ad1c7878ff4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:36.108499 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad5ad9f4-7498-416b-9042-0ad1c7878ff4 admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:36.108871 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad5ad9f4-7498-416b-9042-0ad1c7878ff4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:36.109217 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad5ad9f4-7498-416b-9042-0ad1c7878ff4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:36.109945 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad5ad9f4-7498-416b-9042-0ad1c7878ff4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:36.111740 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad5ad9f4-7498-416b-9042-0ad1c7878ff4 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:36.122571 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1152/2275] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:13:36 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:13:36.126964 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39d94abb-1310-4a34-8fb1-082a796a441c admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:36.143293 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39d94abb-1310-4a34-8fb1-082a796a441c admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:36.148153 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1578a71-5a1f-4fe9-8964-4759e9e05f53 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:36.148861 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1578a71-5a1f-4fe9-8964-4759e9e05f53 admin admin] REQUEST_METHOD: `PUT` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:36.149140 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1578a71-5a1f-4fe9-8964-4759e9e05f53 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:36.149411 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1578a71-5a1f-4fe9-8964-4759e9e05f53 admin admin] PATH_INFO: `/v3/projects/6bef7d3759a248a7a76b5b5d9a4a2880/users/b85679e0df3f47dc995a5437a41f037f/roles/9d9d5d2c270146488249b7ed943280db` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:36.165986 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1578a71-5a1f-4fe9-8964-4759e9e05f53 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6bef7d3759a248a7a76b5b5d9a4a2880, user_id=b85679e0df3f47dc995a5437a41f037f, role_id=9d9d5d2c270146488249b7ed943280db)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:36.168693 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1578a71-5a1f-4fe9-8964-4759e9e05f53 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:36.183140 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1124/2276] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:13:36 2022] PUT /identity/v3/projects/6bef7d3759a248a7a76b5b5d9a4a2880/users/b85679e0df3f47dc995a5437a41f037f/roles/9d9d5d2c270146488249b7ed943280db => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:13:36.189998 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad5ad9f4-7498-416b-9042-0ad1c7878ff4 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:36.199843 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad5ad9f4-7498-416b-9042-0ad1c7878ff4 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:36.212428 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:36.257098 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-369acfab-7495-4ccb-89d1-30a5a12b7d4c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:36.257694 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-369acfab-7495-4ccb-89d1-30a5a12b7d4c admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:36.257970 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-369acfab-7495-4ccb-89d1-30a5a12b7d4c admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:36.258239 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-369acfab-7495-4ccb-89d1-30a5a12b7d4c admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:36.258906 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-369acfab-7495-4ccb-89d1-30a5a12b7d4c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:36.260487 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-369acfab-7495-4ccb-89d1-30a5a12b7d4c admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:36.271187 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1153/2277] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:13:36 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:13:36.275495 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1578a71-5a1f-4fe9-8964-4759e9e05f53 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:36.286295 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1578a71-5a1f-4fe9-8964-4759e9e05f53 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:36.290911 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d4c4dae-5129-4ebf-8623-ac9df091fc57 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:36.291544 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d4c4dae-5129-4ebf-8623-ac9df091fc57 admin admin] REQUEST_METHOD: `PUT` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:36.291822 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d4c4dae-5129-4ebf-8623-ac9df091fc57 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:36.292090 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d4c4dae-5129-4ebf-8623-ac9df091fc57 admin admin] PATH_INFO: `/v3/projects/6bef7d3759a248a7a76b5b5d9a4a2880/users/b85679e0df3f47dc995a5437a41f037f/roles/9d9d5d2c270146488249b7ed943280db` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:36.293973 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d4c4dae-5129-4ebf-8623-ac9df091fc57 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6bef7d3759a248a7a76b5b5d9a4a2880, user_id=b85679e0df3f47dc995a5437a41f037f, role_id=9d9d5d2c270146488249b7ed943280db)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:36.296630 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d4c4dae-5129-4ebf-8623-ac9df091fc57 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:36.306527 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1125/2278] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:13:36 2022] PUT /identity/v3/projects/6bef7d3759a248a7a76b5b5d9a4a2880/users/b85679e0df3f47dc995a5437a41f037f/roles/9d9d5d2c270146488249b7ed943280db => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:13:36.317362 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22542ddc-515e-476b-9f72-9abb23a85d7f None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:36.317689 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22542ddc-515e-476b-9f72-9abb23a85d7f None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:36.317958 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22542ddc-515e-476b-9f72-9abb23a85d7f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:36.346286 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-22542ddc-515e-476b-9f72-9abb23a85d7f None None] MFA Rules not processed for user `b85679e0df3f47dc995a5437a41f037f`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:13:36.347442 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-22542ddc-515e-476b-9f72-9abb23a85d7f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:36.414103 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1154/2279] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:13:36 2022] POST /identity/v3/auth/tokens => generated 3337 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 05:13:36.424563 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d4c4dae-5129-4ebf-8623-ac9df091fc57 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:36.426229 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-1d4c4dae-5129-4ebf-8623-ac9df091fc57 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:36.490102 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d4c4dae-5129-4ebf-8623-ac9df091fc57 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:36.493769 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31eb9a9b-08cf-47b0-93a0-55420cfe7904 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '019a79b2affe478b8a30d1c538d0415c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:36.494368 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31eb9a9b-08cf-47b0-93a0-55420cfe7904 service cinder] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:36.494679 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31eb9a9b-08cf-47b0-93a0-55420cfe7904 service cinder] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:36.496497 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31eb9a9b-08cf-47b0-93a0-55420cfe7904 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:36.506456 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31eb9a9b-08cf-47b0-93a0-55420cfe7904 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:36.507916 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31eb9a9b-08cf-47b0-93a0-55420cfe7904 service cinder] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:36.518077 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1126/2280] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:13:36 2022] GET /identity/v3/auth/tokens => generated 3337 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:13:36.562455 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22542ddc-515e-476b-9f72-9abb23a85d7f None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:36.563870 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-22542ddc-515e-476b-9f72-9abb23a85d7f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:36.576268 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22542ddc-515e-476b-9f72-9abb23a85d7f None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:36.579775 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7856ddb3-9807-484b-b3d2-e59be044b0d8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '019a79b2affe478b8a30d1c538d0415c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:36.581509 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7856ddb3-9807-484b-b3d2-e59be044b0d8 service cinder] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:36.581779 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7856ddb3-9807-484b-b3d2-e59be044b0d8 service cinder] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:36.582041 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7856ddb3-9807-484b-b3d2-e59be044b0d8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:36.637984 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7856ddb3-9807-484b-b3d2-e59be044b0d8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:36.639440 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7856ddb3-9807-484b-b3d2-e59be044b0d8 service cinder] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:36.648974 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1155/2281] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:13:36 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:13:40.008610 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31eb9a9b-08cf-47b0-93a0-55420cfe7904 service cinder] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:40.010233 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-31eb9a9b-08cf-47b0-93a0-55420cfe7904 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:40.125572 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31eb9a9b-08cf-47b0-93a0-55420cfe7904 service cinder] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:40.131718 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa722629-32c0-41df-ba2b-798e524f22b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:40.133560 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa722629-32c0-41df-ba2b-798e524f22b1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:40.133858 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa722629-32c0-41df-ba2b-798e524f22b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:40.134517 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa722629-32c0-41df-ba2b-798e524f22b1 admin admin] PATH_INFO: `/v3/users/9b3c8e44ba2b42bd8a61d1823e8a2744` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:40.136330 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa722629-32c0-41df-ba2b-798e524f22b1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9b3c8e44ba2b42bd8a61d1823e8a2744)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:40.138801 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa722629-32c0-41df-ba2b-798e524f22b1 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:40.221787 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=89,keystone:INSERT=5,keystone:DELETE=6 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:40.229674 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-fa722629-32c0-41df-ba2b-798e524f22b1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9b3c8e44ba2b42bd8a61d1823e8a2744'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:40.234705 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-fa722629-32c0-41df-ba2b-798e524f22b1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9b3c8e44ba2b42bd8a61d1823e8a2744'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:40.235714 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-fa722629-32c0-41df-ba2b-798e524f22b1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9b3c8e44ba2b42bd8a61d1823e8a2744'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:40.263742 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-fa722629-32c0-41df-ba2b-798e524f22b1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9b3c8e44ba2b42bd8a61d1823e8a2744'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:40.271648 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-fa722629-32c0-41df-ba2b-798e524f22b1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9b3c8e44ba2b42bd8a61d1823e8a2744'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:40.295413 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1127/2282] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:13:40 2022] DELETE /identity/v3/users/9b3c8e44ba2b42bd8a61d1823e8a2744 => generated 0 bytes in 287 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:13:40.299755 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-923f8a8c-d571-4170-9be9-7f891eec861d None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:40.300272 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-923f8a8c-d571-4170-9be9-7f891eec861d None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:40.300354 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-923f8a8c-d571-4170-9be9-7f891eec861d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:40.329785 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-923f8a8c-d571-4170-9be9-7f891eec861d None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:13:40.331610 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-923f8a8c-d571-4170-9be9-7f891eec861d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:40.336721 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa722629-32c0-41df-ba2b-798e524f22b1 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:40.338129 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-fa722629-32c0-41df-ba2b-798e524f22b1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:40.417072 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:40.429194 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa722629-32c0-41df-ba2b-798e524f22b1 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:40.437057 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cb26a40-573f-4bef-96a0-7ee155f5c31f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:40.437726 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cb26a40-573f-4bef-96a0-7ee155f5c31f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:40.438174 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cb26a40-573f-4bef-96a0-7ee155f5c31f admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:40.438456 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cb26a40-573f-4bef-96a0-7ee155f5c31f admin admin] PATH_INFO: `/v3/users/272d9721fee94cb6af129c62809ac08d` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:40.440367 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cb26a40-573f-4bef-96a0-7ee155f5c31f admin admin] RBAC: Authorizing `identity:delete_user(user_id=272d9721fee94cb6af129c62809ac08d)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:40.442701 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cb26a40-573f-4bef-96a0-7ee155f5c31f admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:40.448948 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1156/2283] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:13:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:13:40.460945 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-923f8a8c-d571-4170-9be9-7f891eec861d None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:40.464230 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-923f8a8c-d571-4170-9be9-7f891eec861d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:40.511709 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-3cb26a40-573f-4bef-96a0-7ee155f5c31f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '272d9721fee94cb6af129c62809ac08d'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:40.512429 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-3cb26a40-573f-4bef-96a0-7ee155f5c31f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '272d9721fee94cb6af129c62809ac08d'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:40.513054 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-3cb26a40-573f-4bef-96a0-7ee155f5c31f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '272d9721fee94cb6af129c62809ac08d'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:40.527850 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-3cb26a40-573f-4bef-96a0-7ee155f5c31f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '272d9721fee94cb6af129c62809ac08d'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:40.536407 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-3cb26a40-573f-4bef-96a0-7ee155f5c31f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '272d9721fee94cb6af129c62809ac08d'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:40.542039 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-923f8a8c-d571-4170-9be9-7f891eec861d None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:40.547520 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dbe6597-033f-48e5-803f-9f8d43ad3fa1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:40.548612 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dbe6597-033f-48e5-803f-9f8d43ad3fa1 service neutron] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:40.549089 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dbe6597-033f-48e5-803f-9f8d43ad3fa1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:40.549365 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dbe6597-033f-48e5-803f-9f8d43ad3fa1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:40.551338 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-5dbe6597-033f-48e5-803f-9f8d43ad3fa1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:40.552842 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1128/2284] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:13:40 2022] DELETE /identity/v3/users/272d9721fee94cb6af129c62809ac08d => generated 0 bytes in 216 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:13:40.557276 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-908b67e6-88da-44b6-9586-6be80b5a21c5 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:40.557656 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-908b67e6-88da-44b6-9586-6be80b5a21c5 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:40.558020 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-908b67e6-88da-44b6-9586-6be80b5a21c5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:40.567275 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dbe6597-033f-48e5-803f-9f8d43ad3fa1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:40.568811 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dbe6597-033f-48e5-803f-9f8d43ad3fa1 service neutron] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:40.579195 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-908b67e6-88da-44b6-9586-6be80b5a21c5 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:13:40.579362 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1157/2285] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:13:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 119 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:13:40.580735 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-908b67e6-88da-44b6-9586-6be80b5a21c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:40.622376 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:DELETE=14,keystone:SELECT=84,keystone:INSERT=2 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:40.664437 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1129/2286] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:13:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 05:13:40.686811 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dbe6597-033f-48e5-803f-9f8d43ad3fa1 service neutron] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:40.688404 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-5dbe6597-033f-48e5-803f-9f8d43ad3fa1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:40.703161 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dbe6597-033f-48e5-803f-9f8d43ad3fa1 service neutron] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:40.799841 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08776fd5-3951-461c-86b0-dc11074e6a2c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:40.800909 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08776fd5-3951-461c-86b0-dc11074e6a2c service neutron] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:40.800909 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08776fd5-3951-461c-86b0-dc11074e6a2c service neutron] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:40.801173 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08776fd5-3951-461c-86b0-dc11074e6a2c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:40.815965 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08776fd5-3951-461c-86b0-dc11074e6a2c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:40.817419 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08776fd5-3951-461c-86b0-dc11074e6a2c service neutron] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:40.829940 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1158/2287] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:13:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 143 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:13:40.997302 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-908b67e6-88da-44b6-9586-6be80b5a21c5 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:40.999038 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-908b67e6-88da-44b6-9586-6be80b5a21c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:41.020606 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-908b67e6-88da-44b6-9586-6be80b5a21c5 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:41.026230 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ee5e1aa-7f14-4a52-8d8f-7278d1e26b7d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:41.026955 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee5e1aa-7f14-4a52-8d8f-7278d1e26b7d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:41.027302 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee5e1aa-7f14-4a52-8d8f-7278d1e26b7d admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:41.027566 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee5e1aa-7f14-4a52-8d8f-7278d1e26b7d admin admin] PATH_INFO: `/v3/projects/1d6fb13dad924bdaa2588b516fbc9165` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:41.028707 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ee5e1aa-7f14-4a52-8d8f-7278d1e26b7d admin admin] RBAC: Authorizing `identity:delete_project(project_id=1d6fb13dad924bdaa2588b516fbc9165)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:41.030606 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ee5e1aa-7f14-4a52-8d8f-7278d1e26b7d admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:41.082172 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-7ee5e1aa-7f14-4a52-8d8f-7278d1e26b7d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1d6fb13dad924bdaa2588b516fbc9165', 'request_id': 'req-7ee5e1aa-7f14-4a52-8d8f-7278d1e26b7d'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:41.097207 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-7ee5e1aa-7f14-4a52-8d8f-7278d1e26b7d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1d6fb13dad924bdaa2588b516fbc9165', 'request_id': 'req-7ee5e1aa-7f14-4a52-8d8f-7278d1e26b7d'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:41.105485 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-7ee5e1aa-7f14-4a52-8d8f-7278d1e26b7d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1d6fb13dad924bdaa2588b516fbc9165', 'request_id': 'req-7ee5e1aa-7f14-4a52-8d8f-7278d1e26b7d'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:41.129998 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-7ee5e1aa-7f14-4a52-8d8f-7278d1e26b7d admin admin] The token cache is being invalidate because project 1d6fb13dad924bdaa2588b516fbc9165 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81624) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:13:41.130368 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-7ee5e1aa-7f14-4a52-8d8f-7278d1e26b7d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:41.131222 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-7ee5e1aa-7f14-4a52-8d8f-7278d1e26b7d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:41.133040 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1130/2288] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:13:40 2022] DELETE /identity/v3/projects/1d6fb13dad924bdaa2588b516fbc9165 => generated 0 bytes in 136 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:13:41.138364 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b809fc1-eb47-4735-b830-86fa0a0d481e None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:41.138639 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b809fc1-eb47-4735-b830-86fa0a0d481e None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:41.138953 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b809fc1-eb47-4735-b830-86fa0a0d481e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:41.161206 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-0b809fc1-eb47-4735-b830-86fa0a0d481e None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:13:41.162422 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-0b809fc1-eb47-4735-b830-86fa0a0d481e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:41.246484 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1159/2289] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:13:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 05:13:41.252820 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ee5e1aa-7f14-4a52-8d8f-7278d1e26b7d admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:41.254635 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-7ee5e1aa-7f14-4a52-8d8f-7278d1e26b7d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:41.255439 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4efd5d88-31ab-4447-96ce-5e3ee1a69541 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:41.255857 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4efd5d88-31ab-4447-96ce-5e3ee1a69541 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:41.256372 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4efd5d88-31ab-4447-96ce-5e3ee1a69541 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:41.267904 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ee5e1aa-7f14-4a52-8d8f-7278d1e26b7d admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:41.272542 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a309e98e-560c-4e6e-a342-edc118c89a2a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:41.273169 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a309e98e-560c-4e6e-a342-edc118c89a2a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:41.273452 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a309e98e-560c-4e6e-a342-edc118c89a2a admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:41.273725 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a309e98e-560c-4e6e-a342-edc118c89a2a admin admin] PATH_INFO: `/v3/projects/e911a926e3314cdebb9633139261dcbd` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:41.274905 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a309e98e-560c-4e6e-a342-edc118c89a2a admin admin] RBAC: Authorizing `identity:delete_project(project_id=e911a926e3314cdebb9633139261dcbd)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:41.276812 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a309e98e-560c-4e6e-a342-edc118c89a2a admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:41.294771 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-4efd5d88-31ab-4447-96ce-5e3ee1a69541 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:13:41.296212 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-4efd5d88-31ab-4447-96ce-5e3ee1a69541 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:41.304821 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1160/2290] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:13:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 51 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 05:13:41.309137 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4efd5d88-31ab-4447-96ce-5e3ee1a69541 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:41.318633 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4efd5d88-31ab-4447-96ce-5e3ee1a69541 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:41.331160 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-a309e98e-560c-4e6e-a342-edc118c89a2a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e911a926e3314cdebb9633139261dcbd', 'request_id': 'req-a309e98e-560c-4e6e-a342-edc118c89a2a'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:41.331598 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:41.340088 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-a309e98e-560c-4e6e-a342-edc118c89a2a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e911a926e3314cdebb9633139261dcbd', 'request_id': 'req-a309e98e-560c-4e6e-a342-edc118c89a2a'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:41.347753 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-a309e98e-560c-4e6e-a342-edc118c89a2a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e911a926e3314cdebb9633139261dcbd', 'request_id': 'req-a309e98e-560c-4e6e-a342-edc118c89a2a'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:41.359452 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-a309e98e-560c-4e6e-a342-edc118c89a2a admin admin] The token cache is being invalidate because project e911a926e3314cdebb9633139261dcbd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81624) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:13:41.359741 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-a309e98e-560c-4e6e-a342-edc118c89a2a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:41.360400 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-a309e98e-560c-4e6e-a342-edc118c89a2a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:41.361987 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1131/2291] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:13:41 2022] DELETE /identity/v3/projects/e911a926e3314cdebb9633139261dcbd => generated 0 bytes in 110 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:13:41.365845 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a309e98e-560c-4e6e-a342-edc118c89a2a admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:41.367217 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-a309e98e-560c-4e6e-a342-edc118c89a2a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:41.380912 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1500e4b3-85f1-4c61-b871-9f459b70e4a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:41.381518 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1500e4b3-85f1-4c61-b871-9f459b70e4a4 admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:41.381796 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1500e4b3-85f1-4c61-b871-9f459b70e4a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:41.382080 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1500e4b3-85f1-4c61-b871-9f459b70e4a4 admin admin] PATH_INFO: `/v3/domains` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:41.382719 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1500e4b3-85f1-4c61-b871-9f459b70e4a4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:13:41.383055 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1500e4b3-85f1-4c61-b871-9f459b70e4a4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:41.384642 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1500e4b3-85f1-4c61-b871-9f459b70e4a4 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:41.404250 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1161/2292] 10.222.0.77 () {58 vars in 1153 bytes} [Tue Aug 30 05:13:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 95 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 05:13:41.405540 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1500e4b3-85f1-4c61-b871-9f459b70e4a4 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:41.411324 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-1500e4b3-85f1-4c61-b871-9f459b70e4a4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:41.428734 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1500e4b3-85f1-4c61-b871-9f459b70e4a4 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:41.431817 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:41.434522 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd4b570e-a360-46a0-965e-1e85344a6865 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:41.435213 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd4b570e-a360-46a0-965e-1e85344a6865 admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:41.435495 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd4b570e-a360-46a0-965e-1e85344a6865 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:41.435767 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd4b570e-a360-46a0-965e-1e85344a6865 admin admin] PATH_INFO: `/v3/projects` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:41.436390 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd4b570e-a360-46a0-965e-1e85344a6865 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:41.438508 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd4b570e-a360-46a0-965e-1e85344a6865 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:41.454239 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a309e98e-560c-4e6e-a342-edc118c89a2a admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:41.461980 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1486bf4f-718f-4b7c-bffe-29850f5b8b2d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:41.463125 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1486bf4f-718f-4b7c-bffe-29850f5b8b2d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:41.463619 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1486bf4f-718f-4b7c-bffe-29850f5b8b2d admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:41.464602 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1486bf4f-718f-4b7c-bffe-29850f5b8b2d admin admin] PATH_INFO: `/v3/users/b85679e0df3f47dc995a5437a41f037f` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:41.466108 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1486bf4f-718f-4b7c-bffe-29850f5b8b2d admin admin] RBAC: Authorizing `identity:delete_user(user_id=b85679e0df3f47dc995a5437a41f037f)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:41.468401 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1486bf4f-718f-4b7c-bffe-29850f5b8b2d admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:41.476777 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1162/2293] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:13:41 2022] POST /identity/v3/projects => generated 375 bytes in 72 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:13:41.481051 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd4b570e-a360-46a0-965e-1e85344a6865 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:41.520319 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-1486bf4f-718f-4b7c-bffe-29850f5b8b2d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b85679e0df3f47dc995a5437a41f037f'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:41.521020 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-1486bf4f-718f-4b7c-bffe-29850f5b8b2d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b85679e0df3f47dc995a5437a41f037f'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:41.521692 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-1486bf4f-718f-4b7c-bffe-29850f5b8b2d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b85679e0df3f47dc995a5437a41f037f'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:41.536638 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-1486bf4f-718f-4b7c-bffe-29850f5b8b2d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b85679e0df3f47dc995a5437a41f037f'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:41.544375 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-1486bf4f-718f-4b7c-bffe-29850f5b8b2d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b85679e0df3f47dc995a5437a41f037f'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:41.554339 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd4b570e-a360-46a0-965e-1e85344a6865 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:41.559687 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71fe4a0c-8c10-44a6-b2dd-de9ad6abfc3c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:41.560352 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71fe4a0c-8c10-44a6-b2dd-de9ad6abfc3c admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:41.560637 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71fe4a0c-8c10-44a6-b2dd-de9ad6abfc3c admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:41.560912 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71fe4a0c-8c10-44a6-b2dd-de9ad6abfc3c admin admin] PATH_INFO: `/v3/users` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:41.564171 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1132/2294] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:13:41 2022] DELETE /identity/v3/users/b85679e0df3f47dc995a5437a41f037f => generated 0 bytes in 196 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 05:13:41.564308 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71fe4a0c-8c10-44a6-b2dd-de9ad6abfc3c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:41.564308 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71fe4a0c-8c10-44a6-b2dd-de9ad6abfc3c admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:41.602020 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1163/2295] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:13:41 2022] POST /identity/v3/users => generated 288 bytes in 118 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 05:13:41.603306 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1486bf4f-718f-4b7c-bffe-29850f5b8b2d admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:41.604867 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-1486bf4f-718f-4b7c-bffe-29850f5b8b2d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:41.730984 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1486bf4f-718f-4b7c-bffe-29850f5b8b2d admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:41.775640 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e1dfb8d-a8da-4e3b-b512-32e6ff34ff97 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:41.778413 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1dfb8d-a8da-4e3b-b512-32e6ff34ff97 admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:41.778912 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1dfb8d-a8da-4e3b-b512-32e6ff34ff97 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:41.779117 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1dfb8d-a8da-4e3b-b512-32e6ff34ff97 admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:41.779795 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e1dfb8d-a8da-4e3b-b512-32e6ff34ff97 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:41.784786 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e1dfb8d-a8da-4e3b-b512-32e6ff34ff97 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:41.802111 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1133/2296] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:13:41 2022] GET /identity/v3/roles => generated 1420 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 05:13:41.806201 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71fe4a0c-8c10-44a6-b2dd-de9ad6abfc3c admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:41.817424 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71fe4a0c-8c10-44a6-b2dd-de9ad6abfc3c admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:41.823951 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df4c08cd-2a91-4c1a-b152-bd8546ed881a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:41.823951 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df4c08cd-2a91-4c1a-b152-bd8546ed881a admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:41.823951 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df4c08cd-2a91-4c1a-b152-bd8546ed881a admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:41.824286 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df4c08cd-2a91-4c1a-b152-bd8546ed881a admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:41.824734 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df4c08cd-2a91-4c1a-b152-bd8546ed881a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:41.826365 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df4c08cd-2a91-4c1a-b152-bd8546ed881a admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:41.839043 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1164/2297] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:13:41 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:13:41.842995 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e1dfb8d-a8da-4e3b-b512-32e6ff34ff97 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:41.853525 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e1dfb8d-a8da-4e3b-b512-32e6ff34ff97 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:41.859505 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23ecb949-4f80-4b7d-a3ca-f85f21ef5314 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:41.860283 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23ecb949-4f80-4b7d-a3ca-f85f21ef5314 admin admin] REQUEST_METHOD: `PUT` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:41.860570 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23ecb949-4f80-4b7d-a3ca-f85f21ef5314 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:41.860846 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23ecb949-4f80-4b7d-a3ca-f85f21ef5314 admin admin] PATH_INFO: `/v3/projects/f62a938e24c641f8bc1d418239c2bc27/users/054db9eeb5704d62bd37c17e74b8a7c1/roles/6f03d86980c34c36b44a8d6764ff4fda` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:41.878987 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23ecb949-4f80-4b7d-a3ca-f85f21ef5314 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f62a938e24c641f8bc1d418239c2bc27, user_id=054db9eeb5704d62bd37c17e74b8a7c1, role_id=6f03d86980c34c36b44a8d6764ff4fda)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:41.881694 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23ecb949-4f80-4b7d-a3ca-f85f21ef5314 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:41.899603 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1134/2298] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:13:41 2022] PUT /identity/v3/projects/f62a938e24c641f8bc1d418239c2bc27/users/054db9eeb5704d62bd37c17e74b8a7c1/roles/6f03d86980c34c36b44a8d6764ff4fda => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:13:41.909777 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-391936f4-72a6-4b92-a733-3d4cfaecf1ef None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:41.910070 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-391936f4-72a6-4b92-a733-3d4cfaecf1ef None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:41.910353 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-391936f4-72a6-4b92-a733-3d4cfaecf1ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:41.942473 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-391936f4-72a6-4b92-a733-3d4cfaecf1ef None None] MFA Rules not processed for user `054db9eeb5704d62bd37c17e74b8a7c1`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:13:41.943353 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-391936f4-72a6-4b92-a733-3d4cfaecf1ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:42.034704 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23ecb949-4f80-4b7d-a3ca-f85f21ef5314 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:42.040595 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-23ecb949-4f80-4b7d-a3ca-f85f21ef5314 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:42.060578 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:42.074977 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1165/2299] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:13:41 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:13:42.088598 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-391936f4-72a6-4b92-a733-3d4cfaecf1ef None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:42.091283 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-391936f4-72a6-4b92-a733-3d4cfaecf1ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:42.125919 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:42.169527 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23ecb949-4f80-4b7d-a3ca-f85f21ef5314 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:42.174754 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3388a448-b835-42c1-a146-4155ead66a8c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:42.175489 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3388a448-b835-42c1-a146-4155ead66a8c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:42.175867 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3388a448-b835-42c1-a146-4155ead66a8c admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:42.176230 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3388a448-b835-42c1-a146-4155ead66a8c admin admin] PATH_INFO: `/v3/projects/6bef7d3759a248a7a76b5b5d9a4a2880` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:42.176387 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-391936f4-72a6-4b92-a733-3d4cfaecf1ef None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:42.177489 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3388a448-b835-42c1-a146-4155ead66a8c admin admin] RBAC: Authorizing `identity:delete_project(project_id=6bef7d3759a248a7a76b5b5d9a4a2880)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:42.179744 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3388a448-b835-42c1-a146-4155ead66a8c admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:42.181085 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84f131b2-134d-4eb9-8ee6-e4e557142970 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '019a79b2affe478b8a30d1c538d0415c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:42.182118 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f131b2-134d-4eb9-8ee6-e4e557142970 service cinder] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:42.182599 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f131b2-134d-4eb9-8ee6-e4e557142970 service cinder] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:42.182912 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f131b2-134d-4eb9-8ee6-e4e557142970 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:42.192930 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84f131b2-134d-4eb9-8ee6-e4e557142970 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:42.195264 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84f131b2-134d-4eb9-8ee6-e4e557142970 service cinder] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:42.206922 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1166/2300] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:13:42 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 121 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:13:42.248660 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-3388a448-b835-42c1-a146-4155ead66a8c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6bef7d3759a248a7a76b5b5d9a4a2880', 'request_id': 'req-3388a448-b835-42c1-a146-4155ead66a8c'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:42.258474 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-3388a448-b835-42c1-a146-4155ead66a8c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6bef7d3759a248a7a76b5b5d9a4a2880', 'request_id': 'req-3388a448-b835-42c1-a146-4155ead66a8c'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:42.265980 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-3388a448-b835-42c1-a146-4155ead66a8c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6bef7d3759a248a7a76b5b5d9a4a2880', 'request_id': 'req-3388a448-b835-42c1-a146-4155ead66a8c'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:42.282098 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-3388a448-b835-42c1-a146-4155ead66a8c admin admin] The token cache is being invalidate because project 6bef7d3759a248a7a76b5b5d9a4a2880 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81624) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:13:42.282473 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-3388a448-b835-42c1-a146-4155ead66a8c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:42.284144 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-3388a448-b835-42c1-a146-4155ead66a8c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:42.285726 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1135/2301] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:13:42 2022] DELETE /identity/v3/projects/6bef7d3759a248a7a76b5b5d9a4a2880 => generated 0 bytes in 252 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:13:42.291019 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f404117-ced9-4d32-bc32-e7b3368042ea None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:42.291388 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f404117-ced9-4d32-bc32-e7b3368042ea None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:42.292466 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f404117-ced9-4d32-bc32-e7b3368042ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:42.323471 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-4f404117-ced9-4d32-bc32-e7b3368042ea None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:13:42.325278 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-4f404117-ced9-4d32-bc32-e7b3368042ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:42.411946 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1167/2302] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:13:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:13:42.419939 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8871bafb-b104-4f05-882a-a7458b4dea02 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:42.420272 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8871bafb-b104-4f05-882a-a7458b4dea02 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:42.420551 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8871bafb-b104-4f05-882a-a7458b4dea02 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:42.440546 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-8871bafb-b104-4f05-882a-a7458b4dea02 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:13:42.441782 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-8871bafb-b104-4f05-882a-a7458b4dea02 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:42.450173 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1136/2303] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:13:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:13:42.455349 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f404117-ced9-4d32-bc32-e7b3368042ea None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:42.470956 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f404117-ced9-4d32-bc32-e7b3368042ea None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:42.476450 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee94ff43-0935-449d-82a4-e78ee447ec70 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:42.477208 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee94ff43-0935-449d-82a4-e78ee447ec70 admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:42.477487 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee94ff43-0935-449d-82a4-e78ee447ec70 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:42.477784 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee94ff43-0935-449d-82a4-e78ee447ec70 admin admin] PATH_INFO: `/v3/domains` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:42.478964 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee94ff43-0935-449d-82a4-e78ee447ec70 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:13:42.479391 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee94ff43-0935-449d-82a4-e78ee447ec70 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:42.481541 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee94ff43-0935-449d-82a4-e78ee447ec70 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:42.496400 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1168/2304] 10.222.0.77 () {58 vars in 1153 bytes} [Tue Aug 30 05:13:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:13:42.501434 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8871bafb-b104-4f05-882a-a7458b4dea02 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:42.514175 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8871bafb-b104-4f05-882a-a7458b4dea02 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:42.520488 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84d2a803-5961-46d2-98a4-338c83e3f344 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:42.521414 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84d2a803-5961-46d2-98a4-338c83e3f344 admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:42.521414 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84d2a803-5961-46d2-98a4-338c83e3f344 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:42.521829 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84d2a803-5961-46d2-98a4-338c83e3f344 admin admin] PATH_INFO: `/v3/projects` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:42.522527 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84d2a803-5961-46d2-98a4-338c83e3f344 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:42.524540 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84d2a803-5961-46d2-98a4-338c83e3f344 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:42.561283 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1137/2305] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:13:42 2022] POST /identity/v3/projects => generated 401 bytes in 60 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:13:42.566090 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee94ff43-0935-449d-82a4-e78ee447ec70 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:42.580898 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee94ff43-0935-449d-82a4-e78ee447ec70 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:42.648324 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17d7fbcb-bf02-48f4-ba65-f91617d03b28 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:42.648846 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d7fbcb-bf02-48f4-ba65-f91617d03b28 admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:42.649419 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d7fbcb-bf02-48f4-ba65-f91617d03b28 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:42.649711 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d7fbcb-bf02-48f4-ba65-f91617d03b28 admin admin] PATH_INFO: `/v3/users` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:42.650362 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17d7fbcb-bf02-48f4-ba65-f91617d03b28 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:42.652742 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17d7fbcb-bf02-48f4-ba65-f91617d03b28 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:42.674778 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1169/2306] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:13:42 2022] POST /identity/v3/users => generated 301 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:13:42.679087 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84d2a803-5961-46d2-98a4-338c83e3f344 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:42.689855 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84d2a803-5961-46d2-98a4-338c83e3f344 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:42.694666 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0282857-639d-48bf-b552-72e9a9495e9a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:42.695285 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0282857-639d-48bf-b552-72e9a9495e9a admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:42.695563 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0282857-639d-48bf-b552-72e9a9495e9a admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:42.695832 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0282857-639d-48bf-b552-72e9a9495e9a admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:42.696473 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0282857-639d-48bf-b552-72e9a9495e9a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:42.698152 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0282857-639d-48bf-b552-72e9a9495e9a admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:42.709097 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1138/2307] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:13:42 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:13:42.713168 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17d7fbcb-bf02-48f4-ba65-f91617d03b28 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:42.725202 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17d7fbcb-bf02-48f4-ba65-f91617d03b28 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:42.730059 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-561247dd-3ccd-4726-82aa-f47a3938eb32 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:42.730694 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-561247dd-3ccd-4726-82aa-f47a3938eb32 admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:42.731007 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-561247dd-3ccd-4726-82aa-f47a3938eb32 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:42.731282 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-561247dd-3ccd-4726-82aa-f47a3938eb32 admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:42.731924 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-561247dd-3ccd-4726-82aa-f47a3938eb32 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:42.733588 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-561247dd-3ccd-4726-82aa-f47a3938eb32 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:42.748022 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1170/2308] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:13:42 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:13:42.749297 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0282857-639d-48bf-b552-72e9a9495e9a admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:42.760365 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0282857-639d-48bf-b552-72e9a9495e9a admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:42.765788 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8381a534-edfb-4e07-844a-fd7182ac28c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:42.766462 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8381a534-edfb-4e07-844a-fd7182ac28c9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:42.766739 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8381a534-edfb-4e07-844a-fd7182ac28c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:42.767053 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8381a534-edfb-4e07-844a-fd7182ac28c9 admin admin] PATH_INFO: `/v3/projects/e546d038f14c404194fc01642c6d7d69/users/7e2e8cfbe12a49568b54f02bb14bc1e3/roles/6f03d86980c34c36b44a8d6764ff4fda` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:42.782031 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8381a534-edfb-4e07-844a-fd7182ac28c9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e546d038f14c404194fc01642c6d7d69, user_id=7e2e8cfbe12a49568b54f02bb14bc1e3, role_id=6f03d86980c34c36b44a8d6764ff4fda)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:42.784807 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8381a534-edfb-4e07-844a-fd7182ac28c9 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:42.797386 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1139/2309] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:13:42 2022] PUT /identity/v3/projects/e546d038f14c404194fc01642c6d7d69/users/7e2e8cfbe12a49568b54f02bb14bc1e3/roles/6f03d86980c34c36b44a8d6764ff4fda => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:13:42.807696 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41eb6145-171a-467c-8d30-76635981cf9c None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:42.807978 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41eb6145-171a-467c-8d30-76635981cf9c None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:42.808247 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41eb6145-171a-467c-8d30-76635981cf9c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:42.821611 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:42.836829 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-41eb6145-171a-467c-8d30-76635981cf9c None None] MFA Rules not processed for user `7e2e8cfbe12a49568b54f02bb14bc1e3`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:13:42.838018 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-41eb6145-171a-467c-8d30-76635981cf9c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:42.902068 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1171/2310] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:13:42 2022] POST /identity/v3/auth/tokens => generated 3278 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:13:42.906507 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8381a534-edfb-4e07-844a-fd7182ac28c9 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:42.969036 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8381a534-edfb-4e07-844a-fd7182ac28c9 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:42.973668 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6311b7b1-f078-4f76-94d5-87f9623408e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:42.974255 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6311b7b1-f078-4f76-94d5-87f9623408e1 admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:42.974554 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6311b7b1-f078-4f76-94d5-87f9623408e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:42.974911 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6311b7b1-f078-4f76-94d5-87f9623408e1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:42.975529 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6311b7b1-f078-4f76-94d5-87f9623408e1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:42.977460 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6311b7b1-f078-4f76-94d5-87f9623408e1 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:42.987616 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:43.006777 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1140/2311] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:13:42 2022] POST /identity/v3/projects => generated 401 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:13:43.012235 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41eb6145-171a-467c-8d30-76635981cf9c None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:43.022726 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41eb6145-171a-467c-8d30-76635981cf9c None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:43.080198 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c509bf78-39bf-4527-8fe8-74a0995a7105 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:43.080794 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c509bf78-39bf-4527-8fe8-74a0995a7105 admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:43.081069 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c509bf78-39bf-4527-8fe8-74a0995a7105 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:43.081362 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c509bf78-39bf-4527-8fe8-74a0995a7105 admin admin] PATH_INFO: `/v3/users` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:43.081997 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c509bf78-39bf-4527-8fe8-74a0995a7105 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:43.083935 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c509bf78-39bf-4527-8fe8-74a0995a7105 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:43.105814 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1172/2312] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:13:43 2022] POST /identity/v3/users => generated 307 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 05:13:43.109788 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6311b7b1-f078-4f76-94d5-87f9623408e1 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:43.121523 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6311b7b1-f078-4f76-94d5-87f9623408e1 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:43.126102 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4db0e5ab-0750-493a-99ff-83fa771c4bcc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:43.127361 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4db0e5ab-0750-493a-99ff-83fa771c4bcc admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:43.127642 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4db0e5ab-0750-493a-99ff-83fa771c4bcc admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:43.127906 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4db0e5ab-0750-493a-99ff-83fa771c4bcc admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:43.128535 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4db0e5ab-0750-493a-99ff-83fa771c4bcc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:43.130125 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4db0e5ab-0750-493a-99ff-83fa771c4bcc admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:43.140457 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1141/2313] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:13:43 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:13:43.144601 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c509bf78-39bf-4527-8fe8-74a0995a7105 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:43.155255 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c509bf78-39bf-4527-8fe8-74a0995a7105 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:43.160007 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d14bcd14-91ea-4651-adaf-3c92f68fed91 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:43.160732 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d14bcd14-91ea-4651-adaf-3c92f68fed91 admin admin] REQUEST_METHOD: `PUT` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:43.161101 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d14bcd14-91ea-4651-adaf-3c92f68fed91 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:43.161444 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d14bcd14-91ea-4651-adaf-3c92f68fed91 admin admin] PATH_INFO: `/v3/projects/607b1f5d33de45bd93b31ab002707d1a/users/ac3f55be47ae427bb872f45a6248aacb/roles/9d9d5d2c270146488249b7ed943280db` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:43.177548 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d14bcd14-91ea-4651-adaf-3c92f68fed91 admin admin] RBAC: Authorizing `identity:create_grant(project_id=607b1f5d33de45bd93b31ab002707d1a, user_id=ac3f55be47ae427bb872f45a6248aacb, role_id=9d9d5d2c270146488249b7ed943280db)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:43.180332 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d14bcd14-91ea-4651-adaf-3c92f68fed91 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:43.195403 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1173/2314] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:13:43 2022] PUT /identity/v3/projects/607b1f5d33de45bd93b31ab002707d1a/users/ac3f55be47ae427bb872f45a6248aacb/roles/9d9d5d2c270146488249b7ed943280db => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 05:13:43.199463 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4db0e5ab-0750-493a-99ff-83fa771c4bcc admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:43.209701 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4db0e5ab-0750-493a-99ff-83fa771c4bcc admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:43.265777 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcee8dfd-d5df-4f30-a0ee-90b8850f5446 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:43.266379 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcee8dfd-d5df-4f30-a0ee-90b8850f5446 admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:43.266662 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcee8dfd-d5df-4f30-a0ee-90b8850f5446 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:43.266967 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcee8dfd-d5df-4f30-a0ee-90b8850f5446 admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:43.267682 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcee8dfd-d5df-4f30-a0ee-90b8850f5446 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:43.269266 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcee8dfd-d5df-4f30-a0ee-90b8850f5446 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:43.279182 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1142/2315] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:13:43 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:13:43.283852 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d14bcd14-91ea-4651-adaf-3c92f68fed91 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:43.294602 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d14bcd14-91ea-4651-adaf-3c92f68fed91 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:43.300184 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b39b67d9-6a61-44a0-87c7-f0989d9df998 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:43.300920 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b39b67d9-6a61-44a0-87c7-f0989d9df998 admin admin] REQUEST_METHOD: `PUT` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:43.301221 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b39b67d9-6a61-44a0-87c7-f0989d9df998 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:43.302094 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b39b67d9-6a61-44a0-87c7-f0989d9df998 admin admin] PATH_INFO: `/v3/projects/607b1f5d33de45bd93b31ab002707d1a/users/ac3f55be47ae427bb872f45a6248aacb/roles/9d9d5d2c270146488249b7ed943280db` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:43.304156 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b39b67d9-6a61-44a0-87c7-f0989d9df998 admin admin] RBAC: Authorizing `identity:create_grant(project_id=607b1f5d33de45bd93b31ab002707d1a, user_id=ac3f55be47ae427bb872f45a6248aacb, role_id=9d9d5d2c270146488249b7ed943280db)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:43.306862 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b39b67d9-6a61-44a0-87c7-f0989d9df998 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:43.318419 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1174/2316] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:13:43 2022] PUT /identity/v3/projects/607b1f5d33de45bd93b31ab002707d1a/users/ac3f55be47ae427bb872f45a6248aacb/roles/9d9d5d2c270146488249b7ed943280db => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 05:13:43.329483 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a46cdba6-8198-4ccb-abc9-4c89a0acc4fe None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:43.329832 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a46cdba6-8198-4ccb-abc9-4c89a0acc4fe None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:43.330178 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a46cdba6-8198-4ccb-abc9-4c89a0acc4fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:43.362016 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-a46cdba6-8198-4ccb-abc9-4c89a0acc4fe None None] MFA Rules not processed for user `ac3f55be47ae427bb872f45a6248aacb`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:13:43.363393 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-a46cdba6-8198-4ccb-abc9-4c89a0acc4fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:43.434074 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b39b67d9-6a61-44a0-87c7-f0989d9df998 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:43.436257 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-b39b67d9-6a61-44a0-87c7-f0989d9df998 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:43.461999 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1143/2317] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:13:43 2022] POST /identity/v3/auth/tokens => generated 3345 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:13:43.483398 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a46cdba6-8198-4ccb-abc9-4c89a0acc4fe None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:43.484814 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-a46cdba6-8198-4ccb-abc9-4c89a0acc4fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:43.518148 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:43.537986 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b39b67d9-6a61-44a0-87c7-f0989d9df998 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:43.547276 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a46cdba6-8198-4ccb-abc9-4c89a0acc4fe None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:43.551194 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afba9457-d2a2-4d8f-9a06-214735a1e201 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '019a79b2affe478b8a30d1c538d0415c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:43.551807 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afba9457-d2a2-4d8f-9a06-214735a1e201 service cinder] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:43.552086 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afba9457-d2a2-4d8f-9a06-214735a1e201 service cinder] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:43.552358 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afba9457-d2a2-4d8f-9a06-214735a1e201 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:43.555307 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1bc2fcb-bacf-4d5f-8e08-a62ea38292f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:43.556059 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1bc2fcb-bacf-4d5f-8e08-a62ea38292f5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:43.556336 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1bc2fcb-bacf-4d5f-8e08-a62ea38292f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:43.556610 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1bc2fcb-bacf-4d5f-8e08-a62ea38292f5 admin admin] PATH_INFO: `/v3/users/4032879b46694b958677eb2465e4e5f0` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:43.557847 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1bc2fcb-bacf-4d5f-8e08-a62ea38292f5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4032879b46694b958677eb2465e4e5f0)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:43.560274 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1bc2fcb-bacf-4d5f-8e08-a62ea38292f5 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:43.562267 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afba9457-d2a2-4d8f-9a06-214735a1e201 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:43.563733 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afba9457-d2a2-4d8f-9a06-214735a1e201 service cinder] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:43.574884 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1144/2318] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:13:43 2022] GET /identity/v3/auth/tokens => generated 3345 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 05:13:43.642859 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-a1bc2fcb-bacf-4d5f-8e08-a62ea38292f5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4032879b46694b958677eb2465e4e5f0'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:43.649546 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-a1bc2fcb-bacf-4d5f-8e08-a62ea38292f5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4032879b46694b958677eb2465e4e5f0'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:43.667237 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-a1bc2fcb-bacf-4d5f-8e08-a62ea38292f5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4032879b46694b958677eb2465e4e5f0'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:43.674237 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afba9457-d2a2-4d8f-9a06-214735a1e201 service cinder] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:43.678554 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:43.679919 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-a1bc2fcb-bacf-4d5f-8e08-a62ea38292f5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4032879b46694b958677eb2465e4e5f0'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:43.681440 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-a1bc2fcb-bacf-4d5f-8e08-a62ea38292f5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4032879b46694b958677eb2465e4e5f0'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:43.684375 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afba9457-d2a2-4d8f-9a06-214735a1e201 service cinder] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:43.688500 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1175/2319] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:13:43 2022] DELETE /identity/v3/users/4032879b46694b958677eb2465e4e5f0 => generated 0 bytes in 255 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:13:43.691870 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-008bb8f5-ea8d-48a3-9e0e-051d772f9cfa service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '019a79b2affe478b8a30d1c538d0415c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:43.692768 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-008bb8f5-ea8d-48a3-9e0e-051d772f9cfa service cinder] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:43.693052 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-008bb8f5-ea8d-48a3-9e0e-051d772f9cfa service cinder] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:43.693338 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-008bb8f5-ea8d-48a3-9e0e-051d772f9cfa service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:43.693610 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d473cc94-258e-4e00-be54-8308be4c9e4a None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:43.693967 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d473cc94-258e-4e00-be54-8308be4c9e4a None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:43.694439 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d473cc94-258e-4e00-be54-8308be4c9e4a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:43.696150 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-008bb8f5-ea8d-48a3-9e0e-051d772f9cfa service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:43.716525 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-d473cc94-258e-4e00-be54-8308be4c9e4a None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:13:43.717889 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-d473cc94-258e-4e00-be54-8308be4c9e4a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:43.827495 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1176/2320] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:13:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 05:13:43.837327 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d473cc94-258e-4e00-be54-8308be4c9e4a None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:43.840572 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-d473cc94-258e-4e00-be54-8308be4c9e4a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:43.840718 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-008bb8f5-ea8d-48a3-9e0e-051d772f9cfa service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:43.842135 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-008bb8f5-ea8d-48a3-9e0e-051d772f9cfa service cinder] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:43.864570 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1145/2321] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:13:43 2022] GET /identity/v3/auth/tokens => generated 3278 bytes in 190 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 05:13:43.971971 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d473cc94-258e-4e00-be54-8308be4c9e4a None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:43.978793 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d1912ac-4015-45c5-ae5b-7b3b00303726 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:43.980567 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d1912ac-4015-45c5-ae5b-7b3b00303726 service neutron] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:43.980854 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d1912ac-4015-45c5-ae5b-7b3b00303726 service neutron] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:43.981130 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d1912ac-4015-45c5-ae5b-7b3b00303726 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:44.007438 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d1912ac-4015-45c5-ae5b-7b3b00303726 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:44.008876 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d1912ac-4015-45c5-ae5b-7b3b00303726 service neutron] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:44.044879 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1177/2322] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:13:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 208 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:13:44.425524 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-008bb8f5-ea8d-48a3-9e0e-051d772f9cfa service cinder] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:44.427043 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-008bb8f5-ea8d-48a3-9e0e-051d772f9cfa service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:44.448131 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-008bb8f5-ea8d-48a3-9e0e-051d772f9cfa service cinder] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:44.455868 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7abefcfe-d824-40a9-b755-effe8c614ad3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:44.456558 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7abefcfe-d824-40a9-b755-effe8c614ad3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:44.456874 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7abefcfe-d824-40a9-b755-effe8c614ad3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:44.457143 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7abefcfe-d824-40a9-b755-effe8c614ad3 admin admin] PATH_INFO: `/v3/projects/ea69acda2e50431380c9df87e247cb2e` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:44.458503 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7abefcfe-d824-40a9-b755-effe8c614ad3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ea69acda2e50431380c9df87e247cb2e)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:44.460986 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7abefcfe-d824-40a9-b755-effe8c614ad3 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:44.551423 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-7abefcfe-d824-40a9-b755-effe8c614ad3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ea69acda2e50431380c9df87e247cb2e', 'request_id': 'req-7abefcfe-d824-40a9-b755-effe8c614ad3'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:44.560857 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-7abefcfe-d824-40a9-b755-effe8c614ad3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ea69acda2e50431380c9df87e247cb2e', 'request_id': 'req-7abefcfe-d824-40a9-b755-effe8c614ad3'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:44.572523 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-7abefcfe-d824-40a9-b755-effe8c614ad3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ea69acda2e50431380c9df87e247cb2e', 'request_id': 'req-7abefcfe-d824-40a9-b755-effe8c614ad3'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:44.586965 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-7abefcfe-d824-40a9-b755-effe8c614ad3 admin admin] The token cache is being invalidate because project ea69acda2e50431380c9df87e247cb2e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81624) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:13:44.588350 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-7abefcfe-d824-40a9-b755-effe8c614ad3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:44.589555 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-7abefcfe-d824-40a9-b755-effe8c614ad3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:44.596452 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1146/2323] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:13:44 2022] DELETE /identity/v3/projects/ea69acda2e50431380c9df87e247cb2e => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:13:44.597551 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc36534-f3bb-404e-9ce2-529fe44ddc00 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:44.597998 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc36534-f3bb-404e-9ce2-529fe44ddc00 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:44.598951 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc36534-f3bb-404e-9ce2-529fe44ddc00 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:44.622457 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-8dc36534-f3bb-404e-9ce2-529fe44ddc00 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:13:44.623710 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-8dc36534-f3bb-404e-9ce2-529fe44ddc00 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:44.629714 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:44.689759 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1178/2324] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:13:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:13:44.697939 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c4f6120-6eb5-4492-8b50-7764574ef3fc None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:44.698234 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c4f6120-6eb5-4492-8b50-7764574ef3fc None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:44.698556 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c4f6120-6eb5-4492-8b50-7764574ef3fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:44.719087 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-1c4f6120-6eb5-4492-8b50-7764574ef3fc None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:13:44.721025 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-1c4f6120-6eb5-4492-8b50-7764574ef3fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:44.729194 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1147/2325] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:13:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:13:44.733595 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dc36534-f3bb-404e-9ce2-529fe44ddc00 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:44.743290 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dc36534-f3bb-404e-9ce2-529fe44ddc00 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:44.747900 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b871c24-ee6c-4274-88b2-eedba05a93f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:44.748496 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b871c24-ee6c-4274-88b2-eedba05a93f6 admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:44.748776 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b871c24-ee6c-4274-88b2-eedba05a93f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:44.749050 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b871c24-ee6c-4274-88b2-eedba05a93f6 admin admin] PATH_INFO: `/v3/domains` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:44.749683 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b871c24-ee6c-4274-88b2-eedba05a93f6 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:13:44.749982 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b871c24-ee6c-4274-88b2-eedba05a93f6 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:44.752171 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b871c24-ee6c-4274-88b2-eedba05a93f6 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:44.765831 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1179/2326] 10.222.0.77 () {58 vars in 1153 bytes} [Tue Aug 30 05:13:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 05:13:44.770705 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c4f6120-6eb5-4492-8b50-7764574ef3fc None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:44.780287 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c4f6120-6eb5-4492-8b50-7764574ef3fc None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:44.785316 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ce3e6aa-a405-4ddf-954d-d22af26a9e28 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:44.785986 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce3e6aa-a405-4ddf-954d-d22af26a9e28 admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:44.786326 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce3e6aa-a405-4ddf-954d-d22af26a9e28 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:44.786591 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce3e6aa-a405-4ddf-954d-d22af26a9e28 admin admin] PATH_INFO: `/v3/projects` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:44.787356 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ce3e6aa-a405-4ddf-954d-d22af26a9e28 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:44.789636 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ce3e6aa-a405-4ddf-954d-d22af26a9e28 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:44.820221 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1148/2327] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:13:44 2022] POST /identity/v3/projects => generated 381 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:13:44.824600 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b871c24-ee6c-4274-88b2-eedba05a93f6 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:44.835109 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b871c24-ee6c-4274-88b2-eedba05a93f6 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:44.910533 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c481ff1-3da3-4af4-9324-880aea567819 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:44.911197 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c481ff1-3da3-4af4-9324-880aea567819 admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:44.911477 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c481ff1-3da3-4af4-9324-880aea567819 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:44.911747 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c481ff1-3da3-4af4-9324-880aea567819 admin admin] PATH_INFO: `/v3/users` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:44.912374 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c481ff1-3da3-4af4-9324-880aea567819 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:44.914289 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c481ff1-3da3-4af4-9324-880aea567819 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:44.938242 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1180/2328] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:13:44 2022] POST /identity/v3/users => generated 291 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 05:13:44.940540 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ce3e6aa-a405-4ddf-954d-d22af26a9e28 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:44.951031 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ce3e6aa-a405-4ddf-954d-d22af26a9e28 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:44.956455 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6af4ec3-1c37-41a9-8f21-c64f407973a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:44.957065 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6af4ec3-1c37-41a9-8f21-c64f407973a3 admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:44.957328 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6af4ec3-1c37-41a9-8f21-c64f407973a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:44.957601 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6af4ec3-1c37-41a9-8f21-c64f407973a3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:44.958237 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6af4ec3-1c37-41a9-8f21-c64f407973a3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:44.959867 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6af4ec3-1c37-41a9-8f21-c64f407973a3 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:44.970113 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1149/2329] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:13:44 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:13:44.973923 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c481ff1-3da3-4af4-9324-880aea567819 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:44.984110 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c481ff1-3da3-4af4-9324-880aea567819 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:44.988754 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28c970db-694d-4035-9e5a-19f0209e19b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:44.989319 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28c970db-694d-4035-9e5a-19f0209e19b3 admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:44.989592 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28c970db-694d-4035-9e5a-19f0209e19b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:44.989857 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28c970db-694d-4035-9e5a-19f0209e19b3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:44.990485 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28c970db-694d-4035-9e5a-19f0209e19b3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:44.992165 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28c970db-694d-4035-9e5a-19f0209e19b3 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:45.002989 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1181/2330] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:13:44 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:13:45.007115 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6af4ec3-1c37-41a9-8f21-c64f407973a3 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:45.017038 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6af4ec3-1c37-41a9-8f21-c64f407973a3 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:45.022246 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72bfeb93-c9d6-4622-8079-b0f0abab2f30 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:45.022909 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72bfeb93-c9d6-4622-8079-b0f0abab2f30 admin admin] REQUEST_METHOD: `PUT` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:45.023186 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72bfeb93-c9d6-4622-8079-b0f0abab2f30 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:45.023459 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72bfeb93-c9d6-4622-8079-b0f0abab2f30 admin admin] PATH_INFO: `/v3/projects/d98f56be5fb14ebe9e938ebc0f642994/users/7522bcceac404e8aa10f719507fe1436/roles/6f03d86980c34c36b44a8d6764ff4fda` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:45.037437 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72bfeb93-c9d6-4622-8079-b0f0abab2f30 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d98f56be5fb14ebe9e938ebc0f642994, user_id=7522bcceac404e8aa10f719507fe1436, role_id=6f03d86980c34c36b44a8d6764ff4fda)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:45.040112 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72bfeb93-c9d6-4622-8079-b0f0abab2f30 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:45.053838 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1150/2331] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:13:45 2022] PUT /identity/v3/projects/d98f56be5fb14ebe9e938ebc0f642994/users/7522bcceac404e8aa10f719507fe1436/roles/6f03d86980c34c36b44a8d6764ff4fda => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:13:45.059042 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49879f53-ecfd-4e3e-9bd1-0cbe5f2a31da None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:45.059341 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49879f53-ecfd-4e3e-9bd1-0cbe5f2a31da None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:45.059662 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49879f53-ecfd-4e3e-9bd1-0cbe5f2a31da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:45.078855 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-49879f53-ecfd-4e3e-9bd1-0cbe5f2a31da None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:13:45.080687 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-49879f53-ecfd-4e3e-9bd1-0cbe5f2a31da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:45.143171 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1182/2332] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:13:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:13:45.152697 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72bfeb93-c9d6-4622-8079-b0f0abab2f30 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:45.154396 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-72bfeb93-c9d6-4622-8079-b0f0abab2f30 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:45.177242 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:45.215956 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72bfeb93-c9d6-4622-8079-b0f0abab2f30 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:45.219959 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c986af8f-fb1b-4fd9-8a55-80f0af1eca3d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:45.220950 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c986af8f-fb1b-4fd9-8a55-80f0af1eca3d service neutron] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:45.221234 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c986af8f-fb1b-4fd9-8a55-80f0af1eca3d service neutron] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:45.221512 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c986af8f-fb1b-4fd9-8a55-80f0af1eca3d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:45.232331 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c986af8f-fb1b-4fd9-8a55-80f0af1eca3d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:45.233811 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c986af8f-fb1b-4fd9-8a55-80f0af1eca3d service neutron] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:45.244001 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1151/2333] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:13:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:13:45.579183 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49879f53-ecfd-4e3e-9bd1-0cbe5f2a31da None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:45.580740 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-49879f53-ecfd-4e3e-9bd1-0cbe5f2a31da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:45.603470 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49879f53-ecfd-4e3e-9bd1-0cbe5f2a31da None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:45.611617 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4fbe94e-bf3d-48db-9aea-f3bf9008263f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:45.612839 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4fbe94e-bf3d-48db-9aea-f3bf9008263f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:45.613126 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4fbe94e-bf3d-48db-9aea-f3bf9008263f admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:45.613820 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4fbe94e-bf3d-48db-9aea-f3bf9008263f admin admin] PATH_INFO: `/v3/users/054db9eeb5704d62bd37c17e74b8a7c1` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:45.614997 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4fbe94e-bf3d-48db-9aea-f3bf9008263f admin admin] RBAC: Authorizing `identity:delete_user(user_id=054db9eeb5704d62bd37c17e74b8a7c1)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:45.617027 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4fbe94e-bf3d-48db-9aea-f3bf9008263f admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:45.633259 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=96,keystone:INSERT=3,keystone:DELETE=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:45.682420 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-b4fbe94e-bf3d-48db-9aea-f3bf9008263f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '054db9eeb5704d62bd37c17e74b8a7c1'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:45.692921 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-b4fbe94e-bf3d-48db-9aea-f3bf9008263f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '054db9eeb5704d62bd37c17e74b8a7c1'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:45.712507 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-b4fbe94e-bf3d-48db-9aea-f3bf9008263f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '054db9eeb5704d62bd37c17e74b8a7c1'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:45.721777 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-b4fbe94e-bf3d-48db-9aea-f3bf9008263f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '054db9eeb5704d62bd37c17e74b8a7c1'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:45.724461 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-b4fbe94e-bf3d-48db-9aea-f3bf9008263f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '054db9eeb5704d62bd37c17e74b8a7c1'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:45.730862 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1183/2334] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:13:45 2022] DELETE /identity/v3/users/054db9eeb5704d62bd37c17e74b8a7c1 => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:13:45.735926 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dd71542-ca77-4a9c-98c9-d8cf94c229fe None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:45.737039 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dd71542-ca77-4a9c-98c9-d8cf94c229fe None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:45.737330 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dd71542-ca77-4a9c-98c9-d8cf94c229fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:45.761063 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-7dd71542-ca77-4a9c-98c9-d8cf94c229fe None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:13:45.762848 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-7dd71542-ca77-4a9c-98c9-d8cf94c229fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:45.911394 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1152/2335] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:13:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:13:45.959824 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4fbe94e-bf3d-48db-9aea-f3bf9008263f admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:45.961370 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-b4fbe94e-bf3d-48db-9aea-f3bf9008263f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:46.094432 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4fbe94e-bf3d-48db-9aea-f3bf9008263f admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:46.099093 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c197f22-277b-4c91-84c6-059178839a8c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:46.099720 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c197f22-277b-4c91-84c6-059178839a8c service neutron] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:46.100000 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c197f22-277b-4c91-84c6-059178839a8c service neutron] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:46.100265 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c197f22-277b-4c91-84c6-059178839a8c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:46.112086 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c197f22-277b-4c91-84c6-059178839a8c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:46.113352 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c197f22-277b-4c91-84c6-059178839a8c service neutron] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:46.126059 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1184/2336] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:13:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:13:46.669534 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a4e4b6-e71c-4efb-ab1d-c9df39462fea None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:46.669841 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a4e4b6-e71c-4efb-ab1d-c9df39462fea None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:46.670216 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a4e4b6-e71c-4efb-ab1d-c9df39462fea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:46.759972 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-c5a4e4b6-e71c-4efb-ab1d-c9df39462fea None None] MFA Rules not processed for user `7522bcceac404e8aa10f719507fe1436`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:13:46.765649 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c197f22-277b-4c91-84c6-059178839a8c service neutron] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:46.767667 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-2c197f22-277b-4c91-84c6-059178839a8c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:46.767786 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-c5a4e4b6-e71c-4efb-ab1d-c9df39462fea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:46.797129 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c197f22-277b-4c91-84c6-059178839a8c service neutron] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:46.812506 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af4ac99e-6da2-4f06-877f-0cdec9bf57ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:46.813281 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af4ac99e-6da2-4f06-877f-0cdec9bf57ee admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:46.813521 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af4ac99e-6da2-4f06-877f-0cdec9bf57ee admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:46.813800 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af4ac99e-6da2-4f06-877f-0cdec9bf57ee admin admin] PATH_INFO: `/v3/projects/f62a938e24c641f8bc1d418239c2bc27` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:46.816891 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af4ac99e-6da2-4f06-877f-0cdec9bf57ee admin admin] RBAC: Authorizing `identity:delete_project(project_id=f62a938e24c641f8bc1d418239c2bc27)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:46.823178 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af4ac99e-6da2-4f06-877f-0cdec9bf57ee admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:46.956575 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-af4ac99e-6da2-4f06-877f-0cdec9bf57ee admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f62a938e24c641f8bc1d418239c2bc27', 'request_id': 'req-af4ac99e-6da2-4f06-877f-0cdec9bf57ee'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:46.971635 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-af4ac99e-6da2-4f06-877f-0cdec9bf57ee admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f62a938e24c641f8bc1d418239c2bc27', 'request_id': 'req-af4ac99e-6da2-4f06-877f-0cdec9bf57ee'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:46.978515 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1153/2337] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:13:46 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 308 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:13:46.999014 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-af4ac99e-6da2-4f06-877f-0cdec9bf57ee admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f62a938e24c641f8bc1d418239c2bc27', 'request_id': 'req-af4ac99e-6da2-4f06-877f-0cdec9bf57ee'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:47.017826 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5a4e4b6-e71c-4efb-ab1d-c9df39462fea None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:47.019412 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-c5a4e4b6-e71c-4efb-ab1d-c9df39462fea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:47.031064 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-af4ac99e-6da2-4f06-877f-0cdec9bf57ee admin admin] The token cache is being invalidate because project f62a938e24c641f8bc1d418239c2bc27 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81625) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:13:47.031416 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-af4ac99e-6da2-4f06-877f-0cdec9bf57ee admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:47.032140 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-af4ac99e-6da2-4f06-877f-0cdec9bf57ee admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:47.034204 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1185/2338] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:13:46 2022] DELETE /identity/v3/projects/f62a938e24c641f8bc1d418239c2bc27 => generated 0 bytes in 269 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 05:13:47.041801 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaa0e6d7-b045-4876-9596-6883bf7d846c None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:47.042170 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaa0e6d7-b045-4876-9596-6883bf7d846c None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:47.042537 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaa0e6d7-b045-4876-9596-6883bf7d846c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:47.096512 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-aaa0e6d7-b045-4876-9596-6883bf7d846c None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:13:47.099347 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-aaa0e6d7-b045-4876-9596-6883bf7d846c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:47.123729 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=82,keystone:DELETE=16,keystone:INSERT=2 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:47.146440 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5a4e4b6-e71c-4efb-ab1d-c9df39462fea None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:47.152261 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:47.198311 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1186/2339] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:13:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:13:47.206955 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb8033c4-afc1-4175-af6c-acc78050db52 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:47.207328 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb8033c4-afc1-4175-af6c-acc78050db52 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:47.207493 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb8033c4-afc1-4175-af6c-acc78050db52 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:47.209351 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7497dcfc-6fed-43e1-b4df-6b9daead7de1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '019a79b2affe478b8a30d1c538d0415c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:47.209965 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7497dcfc-6fed-43e1-b4df-6b9daead7de1 service cinder] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:47.210346 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7497dcfc-6fed-43e1-b4df-6b9daead7de1 service cinder] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:47.210618 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7497dcfc-6fed-43e1-b4df-6b9daead7de1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:47.212409 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-7497dcfc-6fed-43e1-b4df-6b9daead7de1 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:47.231284 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-bb8033c4-afc1-4175-af6c-acc78050db52 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:13:47.232478 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-bb8033c4-afc1-4175-af6c-acc78050db52 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:47.240368 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1187/2340] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:13:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:13:47.244596 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb8033c4-afc1-4175-af6c-acc78050db52 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:47.253720 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb8033c4-afc1-4175-af6c-acc78050db52 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:47.258705 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a910f88-f5c4-40c9-8504-6f617078a6aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:47.259433 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a910f88-f5c4-40c9-8504-6f617078a6aa admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:47.259713 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a910f88-f5c4-40c9-8504-6f617078a6aa admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:47.260109 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a910f88-f5c4-40c9-8504-6f617078a6aa admin admin] PATH_INFO: `/v3/domains` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:47.260753 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a910f88-f5c4-40c9-8504-6f617078a6aa admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:13:47.261190 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a910f88-f5c4-40c9-8504-6f617078a6aa admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:47.262958 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a910f88-f5c4-40c9-8504-6f617078a6aa admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:47.273826 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7497dcfc-6fed-43e1-b4df-6b9daead7de1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:47.275287 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7497dcfc-6fed-43e1-b4df-6b9daead7de1 service cinder] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:47.275647 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1188/2341] 10.222.0.77 () {58 vars in 1153 bytes} [Tue Aug 30 05:13:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 05:13:47.280137 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a910f88-f5c4-40c9-8504-6f617078a6aa admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:47.284302 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1154/2342] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:13:47 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 268 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 05:13:47.290527 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a910f88-f5c4-40c9-8504-6f617078a6aa admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:47.295356 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8672d7c9-7ddf-4518-b4be-9854fbc8ef6f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:47.295935 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8672d7c9-7ddf-4518-b4be-9854fbc8ef6f admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:47.296225 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8672d7c9-7ddf-4518-b4be-9854fbc8ef6f admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:47.296515 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8672d7c9-7ddf-4518-b4be-9854fbc8ef6f admin admin] PATH_INFO: `/v3/projects` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:47.297135 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8672d7c9-7ddf-4518-b4be-9854fbc8ef6f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:47.299144 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8672d7c9-7ddf-4518-b4be-9854fbc8ef6f admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:47.326612 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1189/2343] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:13:47 2022] POST /identity/v3/projects => generated 371 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 05:13:47.331917 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7497dcfc-6fed-43e1-b4df-6b9daead7de1 service cinder] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:47.342640 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7497dcfc-6fed-43e1-b4df-6b9daead7de1 service cinder] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:47.404671 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a111cae4-385b-4624-8263-4a1f2d375f94 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:47.405332 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a111cae4-385b-4624-8263-4a1f2d375f94 admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:47.405707 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a111cae4-385b-4624-8263-4a1f2d375f94 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:47.405875 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a111cae4-385b-4624-8263-4a1f2d375f94 admin admin] PATH_INFO: `/v3/users` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:47.406500 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a111cae4-385b-4624-8263-4a1f2d375f94 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:47.408469 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a111cae4-385b-4624-8263-4a1f2d375f94 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:47.431244 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1155/2344] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:13:47 2022] POST /identity/v3/users => generated 286 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:13:47.435536 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8672d7c9-7ddf-4518-b4be-9854fbc8ef6f admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:47.447462 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8672d7c9-7ddf-4518-b4be-9854fbc8ef6f admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:47.452625 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-479cef8d-3dbe-4030-8640-f0c09985251f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:47.453232 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-479cef8d-3dbe-4030-8640-f0c09985251f admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:47.453519 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-479cef8d-3dbe-4030-8640-f0c09985251f admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:47.453747 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-479cef8d-3dbe-4030-8640-f0c09985251f admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:47.454391 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-479cef8d-3dbe-4030-8640-f0c09985251f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:47.456271 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-479cef8d-3dbe-4030-8640-f0c09985251f admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:47.467123 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1190/2345] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:13:47 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:13:47.471313 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a111cae4-385b-4624-8263-4a1f2d375f94 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:47.481725 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a111cae4-385b-4624-8263-4a1f2d375f94 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:47.486989 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6f19129-7f71-4afc-b574-912d0caa20e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:47.487788 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f19129-7f71-4afc-b574-912d0caa20e2 admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:47.487983 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f19129-7f71-4afc-b574-912d0caa20e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:47.488207 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f19129-7f71-4afc-b574-912d0caa20e2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:47.488859 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6f19129-7f71-4afc-b574-912d0caa20e2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:47.490432 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6f19129-7f71-4afc-b574-912d0caa20e2 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:47.500601 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1156/2346] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:13:47 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:13:47.505559 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-479cef8d-3dbe-4030-8640-f0c09985251f admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:47.516132 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-479cef8d-3dbe-4030-8640-f0c09985251f admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:47.521656 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd4acd72-a6d5-481f-aa2b-7499e8c5a651 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:47.522497 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd4acd72-a6d5-481f-aa2b-7499e8c5a651 admin admin] REQUEST_METHOD: `PUT` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:47.522946 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd4acd72-a6d5-481f-aa2b-7499e8c5a651 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:47.523252 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd4acd72-a6d5-481f-aa2b-7499e8c5a651 admin admin] PATH_INFO: `/v3/projects/156eb098e0fc45af8a02bfde5d69f66f/users/b4386edc28b24181ba71d7d518946b09/roles/6f03d86980c34c36b44a8d6764ff4fda` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:47.539554 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd4acd72-a6d5-481f-aa2b-7499e8c5a651 admin admin] RBAC: Authorizing `identity:create_grant(project_id=156eb098e0fc45af8a02bfde5d69f66f, user_id=b4386edc28b24181ba71d7d518946b09, role_id=6f03d86980c34c36b44a8d6764ff4fda)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:47.542296 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd4acd72-a6d5-481f-aa2b-7499e8c5a651 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:47.559059 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1191/2347] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:13:47 2022] PUT /identity/v3/projects/156eb098e0fc45af8a02bfde5d69f66f/users/b4386edc28b24181ba71d7d518946b09/roles/6f03d86980c34c36b44a8d6764ff4fda => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:13:47.628871 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-803892ce-aaad-47c8-b4d8-37d0551375b2 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:47.629212 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-803892ce-aaad-47c8-b4d8-37d0551375b2 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:47.629444 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-803892ce-aaad-47c8-b4d8-37d0551375b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:47.659349 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-803892ce-aaad-47c8-b4d8-37d0551375b2 None None] MFA Rules not processed for user `b4386edc28b24181ba71d7d518946b09`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:13:47.660575 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-803892ce-aaad-47c8-b4d8-37d0551375b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:47.692302 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:47.727993 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1157/2348] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:13:47 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:13:47.740375 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd4acd72-a6d5-481f-aa2b-7499e8c5a651 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:47.743256 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-dd4acd72-a6d5-481f-aa2b-7499e8c5a651 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:47.807417 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd4acd72-a6d5-481f-aa2b-7499e8c5a651 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:47.811415 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bd4819a-c8a7-48d3-b112-ed31c7b99728 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '019a79b2affe478b8a30d1c538d0415c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:47.812032 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bd4819a-c8a7-48d3-b112-ed31c7b99728 service cinder] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:47.812316 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bd4819a-c8a7-48d3-b112-ed31c7b99728 service cinder] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:47.812590 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bd4819a-c8a7-48d3-b112-ed31c7b99728 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:47.822200 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bd4819a-c8a7-48d3-b112-ed31c7b99728 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:47.824238 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bd4819a-c8a7-48d3-b112-ed31c7b99728 service cinder] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:47.839163 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1192/2349] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:13:47 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:13:47.868219 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-803892ce-aaad-47c8-b4d8-37d0551375b2 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:47.949046 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-803892ce-aaad-47c8-b4d8-37d0551375b2 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:47.953905 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1c366ff-0791-4e6b-a3bd-91fa23982aa5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:47.954566 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1c366ff-0791-4e6b-a3bd-91fa23982aa5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:47.954884 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1c366ff-0791-4e6b-a3bd-91fa23982aa5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:47.955161 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1c366ff-0791-4e6b-a3bd-91fa23982aa5 admin admin] PATH_INFO: `/v3/users/b4386edc28b24181ba71d7d518946b09` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:47.956306 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1c366ff-0791-4e6b-a3bd-91fa23982aa5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b4386edc28b24181ba71d7d518946b09)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:47.958250 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1c366ff-0791-4e6b-a3bd-91fa23982aa5 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:48.025417 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-b1c366ff-0791-4e6b-a3bd-91fa23982aa5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b4386edc28b24181ba71d7d518946b09'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:48.026239 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-b1c366ff-0791-4e6b-a3bd-91fa23982aa5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b4386edc28b24181ba71d7d518946b09'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:48.026879 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-b1c366ff-0791-4e6b-a3bd-91fa23982aa5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b4386edc28b24181ba71d7d518946b09'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:48.040984 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-b1c366ff-0791-4e6b-a3bd-91fa23982aa5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b4386edc28b24181ba71d7d518946b09'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:48.049165 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-b1c366ff-0791-4e6b-a3bd-91fa23982aa5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b4386edc28b24181ba71d7d518946b09'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:48.063781 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1158/2350] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:13:47 2022] DELETE /identity/v3/users/b4386edc28b24181ba71d7d518946b09 => generated 0 bytes in 196 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:13:48.068207 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bffbee0c-598e-4853-b79f-e8a7181030c2 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:48.068487 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bffbee0c-598e-4853-b79f-e8a7181030c2 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:48.068779 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bffbee0c-598e-4853-b79f-e8a7181030c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:48.088004 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-bffbee0c-598e-4853-b79f-e8a7181030c2 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:13:48.089193 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-bffbee0c-598e-4853-b79f-e8a7181030c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:48.152194 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:48.188364 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1193/2351] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:13:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 05:13:48.200734 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1c366ff-0791-4e6b-a3bd-91fa23982aa5 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:48.202158 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-b1c366ff-0791-4e6b-a3bd-91fa23982aa5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:48.295477 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1c366ff-0791-4e6b-a3bd-91fa23982aa5 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:48.301422 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01b0fe8a-4121-4d9b-a81b-4dc025183f55 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:48.302052 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b0fe8a-4121-4d9b-a81b-4dc025183f55 service neutron] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:48.302328 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b0fe8a-4121-4d9b-a81b-4dc025183f55 service neutron] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:48.302670 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b0fe8a-4121-4d9b-a81b-4dc025183f55 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:48.312897 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:48.321252 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01b0fe8a-4121-4d9b-a81b-4dc025183f55 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:48.324828 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01b0fe8a-4121-4d9b-a81b-4dc025183f55 service neutron] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:48.341112 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1159/2352] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:13:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 141 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:13:48.347952 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bffbee0c-598e-4853-b79f-e8a7181030c2 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:48.349680 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-bffbee0c-598e-4853-b79f-e8a7181030c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:48.363261 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bffbee0c-598e-4853-b79f-e8a7181030c2 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:48.368400 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08aadc91-4a45-4bc5-a1f2-102de4891132 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:48.369236 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08aadc91-4a45-4bc5-a1f2-102de4891132 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:48.369611 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08aadc91-4a45-4bc5-a1f2-102de4891132 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:48.370015 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08aadc91-4a45-4bc5-a1f2-102de4891132 admin admin] PATH_INFO: `/v3/users/7e2e8cfbe12a49568b54f02bb14bc1e3` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:48.372051 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08aadc91-4a45-4bc5-a1f2-102de4891132 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7e2e8cfbe12a49568b54f02bb14bc1e3)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:48.374111 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08aadc91-4a45-4bc5-a1f2-102de4891132 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:48.432052 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-08aadc91-4a45-4bc5-a1f2-102de4891132 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7e2e8cfbe12a49568b54f02bb14bc1e3'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:48.445879 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-08aadc91-4a45-4bc5-a1f2-102de4891132 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7e2e8cfbe12a49568b54f02bb14bc1e3'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:48.485030 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-08aadc91-4a45-4bc5-a1f2-102de4891132 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7e2e8cfbe12a49568b54f02bb14bc1e3'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:48.498777 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-08aadc91-4a45-4bc5-a1f2-102de4891132 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7e2e8cfbe12a49568b54f02bb14bc1e3'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:48.500146 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-08aadc91-4a45-4bc5-a1f2-102de4891132 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7e2e8cfbe12a49568b54f02bb14bc1e3'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:48.510944 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1194/2353] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:13:48 2022] DELETE /identity/v3/users/7e2e8cfbe12a49568b54f02bb14bc1e3 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:13:48.514091 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96c3e7e2-c8b5-47d0-aeec-b9f3108391e0 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:48.514348 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96c3e7e2-c8b5-47d0-aeec-b9f3108391e0 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:48.514622 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96c3e7e2-c8b5-47d0-aeec-b9f3108391e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:48.576945 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-96c3e7e2-c8b5-47d0-aeec-b9f3108391e0 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:13:48.578214 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-96c3e7e2-c8b5-47d0-aeec-b9f3108391e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:48.738971 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08aadc91-4a45-4bc5-a1f2-102de4891132 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:48.742389 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-08aadc91-4a45-4bc5-a1f2-102de4891132 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:48.789495 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1160/2354] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:13:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 276 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:13:48.792520 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e1cd1c6-6227-4a09-a227-67a9bbb04712 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:48.793631 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e1cd1c6-6227-4a09-a227-67a9bbb04712 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:48.794027 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e1cd1c6-6227-4a09-a227-67a9bbb04712 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:48.827359 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-7e1cd1c6-6227-4a09-a227-67a9bbb04712 None None] MFA Rules not processed for user `7522bcceac404e8aa10f719507fe1436`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:13:48.828708 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-7e1cd1c6-6227-4a09-a227-67a9bbb04712 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:48.830333 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08aadc91-4a45-4bc5-a1f2-102de4891132 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:48.835268 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b78d0d8-0b51-4659-82e7-9a215ea81c6e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:48.835904 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b78d0d8-0b51-4659-82e7-9a215ea81c6e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:48.836211 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b78d0d8-0b51-4659-82e7-9a215ea81c6e admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:48.836484 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b78d0d8-0b51-4659-82e7-9a215ea81c6e admin admin] PATH_INFO: `/v3/projects/156eb098e0fc45af8a02bfde5d69f66f` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:48.837692 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b78d0d8-0b51-4659-82e7-9a215ea81c6e admin admin] RBAC: Authorizing `identity:delete_project(project_id=156eb098e0fc45af8a02bfde5d69f66f)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:48.839646 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b78d0d8-0b51-4659-82e7-9a215ea81c6e admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:48.894345 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-4b78d0d8-0b51-4659-82e7-9a215ea81c6e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '156eb098e0fc45af8a02bfde5d69f66f', 'request_id': 'req-4b78d0d8-0b51-4659-82e7-9a215ea81c6e'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:48.894932 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1161/2355] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:13:48 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:13:48.895849 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e1cd1c6-6227-4a09-a227-67a9bbb04712 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:48.902476 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-7e1cd1c6-6227-4a09-a227-67a9bbb04712 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:48.907802 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-4b78d0d8-0b51-4659-82e7-9a215ea81c6e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '156eb098e0fc45af8a02bfde5d69f66f', 'request_id': 'req-4b78d0d8-0b51-4659-82e7-9a215ea81c6e'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:48.909987 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=83,keystone:DELETE=15,keystone:INSERT=2 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:48.918945 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-4b78d0d8-0b51-4659-82e7-9a215ea81c6e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '156eb098e0fc45af8a02bfde5d69f66f', 'request_id': 'req-4b78d0d8-0b51-4659-82e7-9a215ea81c6e'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:48.934927 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-4b78d0d8-0b51-4659-82e7-9a215ea81c6e admin admin] The token cache is being invalidate because project 156eb098e0fc45af8a02bfde5d69f66f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81625) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:13:48.935276 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-4b78d0d8-0b51-4659-82e7-9a215ea81c6e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:48.937293 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-4b78d0d8-0b51-4659-82e7-9a215ea81c6e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:48.939701 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1195/2356] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:13:48 2022] DELETE /identity/v3/projects/156eb098e0fc45af8a02bfde5d69f66f => generated 0 bytes in 201 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:13:48.985315 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e1cd1c6-6227-4a09-a227-67a9bbb04712 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:49.071137 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2471931c-bbfd-4a28-b385-5fbdc42af844 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:49.072457 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2471931c-bbfd-4a28-b385-5fbdc42af844 service neutron] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:49.072904 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2471931c-bbfd-4a28-b385-5fbdc42af844 service neutron] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:49.074216 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2471931c-bbfd-4a28-b385-5fbdc42af844 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:49.076817 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-2471931c-bbfd-4a28-b385-5fbdc42af844 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:49.091252 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:49.181037 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2471931c-bbfd-4a28-b385-5fbdc42af844 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:49.182499 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2471931c-bbfd-4a28-b385-5fbdc42af844 service neutron] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:49.195490 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1162/2357] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:13:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 300 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:13:49.510340 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b78d0d8-0b51-4659-82e7-9a215ea81c6e admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:49.511945 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-4b78d0d8-0b51-4659-82e7-9a215ea81c6e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:49.527815 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b78d0d8-0b51-4659-82e7-9a215ea81c6e admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:49.533175 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ed71409-0463-48cf-8175-d76fab2c4d25 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:49.533807 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed71409-0463-48cf-8175-d76fab2c4d25 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:49.534086 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed71409-0463-48cf-8175-d76fab2c4d25 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:49.534418 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed71409-0463-48cf-8175-d76fab2c4d25 admin admin] PATH_INFO: `/v3/projects/e546d038f14c404194fc01642c6d7d69` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:49.535506 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ed71409-0463-48cf-8175-d76fab2c4d25 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e546d038f14c404194fc01642c6d7d69)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:49.538298 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ed71409-0463-48cf-8175-d76fab2c4d25 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:49.609118 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-2ed71409-0463-48cf-8175-d76fab2c4d25 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e546d038f14c404194fc01642c6d7d69', 'request_id': 'req-2ed71409-0463-48cf-8175-d76fab2c4d25'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:49.619018 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-2ed71409-0463-48cf-8175-d76fab2c4d25 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e546d038f14c404194fc01642c6d7d69', 'request_id': 'req-2ed71409-0463-48cf-8175-d76fab2c4d25'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:49.625774 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-2ed71409-0463-48cf-8175-d76fab2c4d25 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e546d038f14c404194fc01642c6d7d69', 'request_id': 'req-2ed71409-0463-48cf-8175-d76fab2c4d25'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:49.637824 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-2ed71409-0463-48cf-8175-d76fab2c4d25 admin admin] The token cache is being invalidate because project e546d038f14c404194fc01642c6d7d69 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81625) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:13:49.638126 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-2ed71409-0463-48cf-8175-d76fab2c4d25 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:49.639119 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-2ed71409-0463-48cf-8175-d76fab2c4d25 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:49.640680 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1196/2358] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:13:49 2022] DELETE /identity/v3/projects/e546d038f14c404194fc01642c6d7d69 => generated 0 bytes in 131 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:13:49.645008 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2471931c-bbfd-4a28-b385-5fbdc42af844 service neutron] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:49.654746 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2471931c-bbfd-4a28-b385-5fbdc42af844 service neutron] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:49.719173 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba419a95-2205-457e-bc90-96b695f7fa38 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:49.719820 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba419a95-2205-457e-bc90-96b695f7fa38 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:49.720173 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba419a95-2205-457e-bc90-96b695f7fa38 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:49.720940 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba419a95-2205-457e-bc90-96b695f7fa38 admin admin] PATH_INFO: `/v3/users/ac3f55be47ae427bb872f45a6248aacb` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:49.722156 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba419a95-2205-457e-bc90-96b695f7fa38 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ac3f55be47ae427bb872f45a6248aacb)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:49.724309 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba419a95-2205-457e-bc90-96b695f7fa38 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:49.779939 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-ba419a95-2205-457e-bc90-96b695f7fa38 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ac3f55be47ae427bb872f45a6248aacb'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:49.780684 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-ba419a95-2205-457e-bc90-96b695f7fa38 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ac3f55be47ae427bb872f45a6248aacb'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:49.781279 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-ba419a95-2205-457e-bc90-96b695f7fa38 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ac3f55be47ae427bb872f45a6248aacb'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:49.795458 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-ba419a95-2205-457e-bc90-96b695f7fa38 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ac3f55be47ae427bb872f45a6248aacb'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:49.802865 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-ba419a95-2205-457e-bc90-96b695f7fa38 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ac3f55be47ae427bb872f45a6248aacb'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:49.818727 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1163/2359] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:13:49 2022] DELETE /identity/v3/users/ac3f55be47ae427bb872f45a6248aacb => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:13:50.315280 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ed71409-0463-48cf-8175-d76fab2c4d25 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:50.317314 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-2ed71409-0463-48cf-8175-d76fab2c4d25 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:50.331937 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ed71409-0463-48cf-8175-d76fab2c4d25 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:50.412395 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2835cab-f640-43dc-ad1b-85ace4c35a81 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:50.413069 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2835cab-f640-43dc-ad1b-85ace4c35a81 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:50.414034 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2835cab-f640-43dc-ad1b-85ace4c35a81 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:50.414323 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2835cab-f640-43dc-ad1b-85ace4c35a81 admin admin] PATH_INFO: `/v3/projects/607b1f5d33de45bd93b31ab002707d1a` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:50.419048 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2835cab-f640-43dc-ad1b-85ace4c35a81 admin admin] RBAC: Authorizing `identity:delete_project(project_id=607b1f5d33de45bd93b31ab002707d1a)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:50.421903 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2835cab-f640-43dc-ad1b-85ace4c35a81 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:50.505452 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-a2835cab-f640-43dc-ad1b-85ace4c35a81 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '607b1f5d33de45bd93b31ab002707d1a', 'request_id': 'req-a2835cab-f640-43dc-ad1b-85ace4c35a81'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:50.515283 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-a2835cab-f640-43dc-ad1b-85ace4c35a81 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '607b1f5d33de45bd93b31ab002707d1a', 'request_id': 'req-a2835cab-f640-43dc-ad1b-85ace4c35a81'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:50.522776 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-a2835cab-f640-43dc-ad1b-85ace4c35a81 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '607b1f5d33de45bd93b31ab002707d1a', 'request_id': 'req-a2835cab-f640-43dc-ad1b-85ace4c35a81'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:50.529751 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=82,keystone:DELETE=16,keystone:INSERT=2 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:50.534725 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-a2835cab-f640-43dc-ad1b-85ace4c35a81 admin admin] The token cache is being invalidate because project 607b1f5d33de45bd93b31ab002707d1a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81625) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:13:50.535128 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-a2835cab-f640-43dc-ad1b-85ace4c35a81 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:50.536067 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-a2835cab-f640-43dc-ad1b-85ace4c35a81 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:50.538238 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1197/2360] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:13:50 2022] DELETE /identity/v3/projects/607b1f5d33de45bd93b31ab002707d1a => generated 0 bytes in 224 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:13:50.543363 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-150cca0f-8af6-4d82-bd4f-7165d20e0a5c None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:50.543881 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-150cca0f-8af6-4d82-bd4f-7165d20e0a5c None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:50.544132 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-150cca0f-8af6-4d82-bd4f-7165d20e0a5c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:50.565694 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-150cca0f-8af6-4d82-bd4f-7165d20e0a5c None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:13:50.567019 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-150cca0f-8af6-4d82-bd4f-7165d20e0a5c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:50.586901 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:50.644275 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1164/2361] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:13:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:13:50.646300 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bbc7f3b-583b-4c43-9238-3c03ac995b1f None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:50.646644 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bbc7f3b-583b-4c43-9238-3c03ac995b1f None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:50.646885 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bbc7f3b-583b-4c43-9238-3c03ac995b1f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:50.666226 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-0bbc7f3b-583b-4c43-9238-3c03ac995b1f None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:13:50.667460 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-0bbc7f3b-583b-4c43-9238-3c03ac995b1f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:50.675877 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1198/2362] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:13:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:13:50.680040 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-150cca0f-8af6-4d82-bd4f-7165d20e0a5c None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:50.692400 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-150cca0f-8af6-4d82-bd4f-7165d20e0a5c None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:50.696916 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7deec37-ae64-448a-bdaa-1bf0794e199e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:50.697505 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7deec37-ae64-448a-bdaa-1bf0794e199e admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:50.697783 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7deec37-ae64-448a-bdaa-1bf0794e199e admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:50.698047 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7deec37-ae64-448a-bdaa-1bf0794e199e admin admin] PATH_INFO: `/v3/domains` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:50.698686 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7deec37-ae64-448a-bdaa-1bf0794e199e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:13:50.699011 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7deec37-ae64-448a-bdaa-1bf0794e199e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:50.700591 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7deec37-ae64-448a-bdaa-1bf0794e199e admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:50.717912 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1165/2363] 10.222.0.77 () {58 vars in 1153 bytes} [Tue Aug 30 05:13:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 05:13:50.725775 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bbc7f3b-583b-4c43-9238-3c03ac995b1f None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:50.735162 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bbc7f3b-583b-4c43-9238-3c03ac995b1f None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:50.740412 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19c98149-3c76-47c0-8211-c223c7809157 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:50.741012 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c98149-3c76-47c0-8211-c223c7809157 admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:50.741303 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c98149-3c76-47c0-8211-c223c7809157 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:50.741549 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c98149-3c76-47c0-8211-c223c7809157 admin admin] PATH_INFO: `/v3/projects` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:50.742176 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19c98149-3c76-47c0-8211-c223c7809157 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:50.744111 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19c98149-3c76-47c0-8211-c223c7809157 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:50.772026 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1199/2364] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:13:50 2022] POST /identity/v3/projects => generated 377 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 05:13:50.776437 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7deec37-ae64-448a-bdaa-1bf0794e199e admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:50.787384 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7deec37-ae64-448a-bdaa-1bf0794e199e admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:50.846967 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08d5dab3-b68f-4c8f-b248-7664ad91df29 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:50.847459 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08d5dab3-b68f-4c8f-b248-7664ad91df29 admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:50.847733 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08d5dab3-b68f-4c8f-b248-7664ad91df29 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:50.848003 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08d5dab3-b68f-4c8f-b248-7664ad91df29 admin admin] PATH_INFO: `/v3/users` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:50.849010 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08d5dab3-b68f-4c8f-b248-7664ad91df29 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:50.850500 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08d5dab3-b68f-4c8f-b248-7664ad91df29 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:50.877436 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1166/2365] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:13:50 2022] POST /identity/v3/users => generated 289 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 05:13:50.878979 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19c98149-3c76-47c0-8211-c223c7809157 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:50.890668 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19c98149-3c76-47c0-8211-c223c7809157 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:50.896375 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7dab004-55c5-42f9-acb1-e3e02d93330c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:50.897128 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7dab004-55c5-42f9-acb1-e3e02d93330c admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:50.897425 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7dab004-55c5-42f9-acb1-e3e02d93330c admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:50.897766 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7dab004-55c5-42f9-acb1-e3e02d93330c admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:50.898412 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7dab004-55c5-42f9-acb1-e3e02d93330c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:50.900154 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7dab004-55c5-42f9-acb1-e3e02d93330c admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:50.914521 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1200/2366] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:13:50 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 05:13:50.916275 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08d5dab3-b68f-4c8f-b248-7664ad91df29 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:50.929521 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08d5dab3-b68f-4c8f-b248-7664ad91df29 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:50.934294 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc0f9fcd-b98c-4600-be8b-1123e417f05a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:50.934936 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc0f9fcd-b98c-4600-be8b-1123e417f05a admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:50.935246 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc0f9fcd-b98c-4600-be8b-1123e417f05a admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:50.935655 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc0f9fcd-b98c-4600-be8b-1123e417f05a admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:50.936306 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc0f9fcd-b98c-4600-be8b-1123e417f05a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:50.937898 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc0f9fcd-b98c-4600-be8b-1123e417f05a admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:50.950198 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1167/2367] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:13:50 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:13:50.954355 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7dab004-55c5-42f9-acb1-e3e02d93330c admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:50.965495 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7dab004-55c5-42f9-acb1-e3e02d93330c admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:50.972297 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-275d8a1c-e1c4-42bf-98dc-5ab43746a9ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:50.973333 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-275d8a1c-e1c4-42bf-98dc-5ab43746a9ea admin admin] REQUEST_METHOD: `PUT` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:50.973333 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-275d8a1c-e1c4-42bf-98dc-5ab43746a9ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:50.973650 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-275d8a1c-e1c4-42bf-98dc-5ab43746a9ea admin admin] PATH_INFO: `/v3/projects/415a364df08041ed9fa305477fadf421/users/3274472d6c9944b395d11d7aa68c664a/roles/6f03d86980c34c36b44a8d6764ff4fda` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:50.993590 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-275d8a1c-e1c4-42bf-98dc-5ab43746a9ea admin admin] RBAC: Authorizing `identity:create_grant(project_id=415a364df08041ed9fa305477fadf421, user_id=3274472d6c9944b395d11d7aa68c664a, role_id=6f03d86980c34c36b44a8d6764ff4fda)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:50.996899 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-275d8a1c-e1c4-42bf-98dc-5ab43746a9ea admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:51.014867 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1201/2368] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:13:50 2022] PUT /identity/v3/projects/415a364df08041ed9fa305477fadf421/users/3274472d6c9944b395d11d7aa68c664a/roles/6f03d86980c34c36b44a8d6764ff4fda => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:13:51.025804 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a0f6dfb-50d2-4919-bdf5-a64dbef42091 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:51.026470 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a0f6dfb-50d2-4919-bdf5-a64dbef42091 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:51.026845 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a0f6dfb-50d2-4919-bdf5-a64dbef42091 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:51.065245 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-8a0f6dfb-50d2-4919-bdf5-a64dbef42091 None None] MFA Rules not processed for user `3274472d6c9944b395d11d7aa68c664a`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:13:51.066590 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-8a0f6dfb-50d2-4919-bdf5-a64dbef42091 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:51.158696 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1168/2369] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:13:51 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:13:51.171141 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-275d8a1c-e1c4-42bf-98dc-5ab43746a9ea admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:51.173537 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-275d8a1c-e1c4-42bf-98dc-5ab43746a9ea admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:51.262014 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-275d8a1c-e1c4-42bf-98dc-5ab43746a9ea admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:51.268009 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ced97c1e-0c54-4590-8888-d959c75f2994 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a002ac0bf240a891ac7304178d3f0e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:51.268509 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ced97c1e-0c54-4590-8888-d959c75f2994 service glance] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:51.269264 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ced97c1e-0c54-4590-8888-d959c75f2994 service glance] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:51.269800 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ced97c1e-0c54-4590-8888-d959c75f2994 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:51.279340 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ced97c1e-0c54-4590-8888-d959c75f2994 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:51.280800 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ced97c1e-0c54-4590-8888-d959c75f2994 service glance] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:51.290795 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1202/2370] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:13:51 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 120 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:13:53.816734 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d42f7c6-7953-459f-a05d-d7fd775c192d None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:53.817609 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d42f7c6-7953-459f-a05d-d7fd775c192d None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:53.818403 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d42f7c6-7953-459f-a05d-d7fd775c192d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:53.853857 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-1d42f7c6-7953-459f-a05d-d7fd775c192d None None] MFA Rules not processed for user `fe51dbec91624ed1ab00b441e29a3dd5`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:13:53.856048 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-1d42f7c6-7953-459f-a05d-d7fd775c192d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:53.881603 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:53.938028 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1169/2371] 10.222.0.77 () {58 vars in 1009 bytes} [Tue Aug 30 05:13:53 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:13:54.004030 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ced97c1e-0c54-4590-8888-d959c75f2994 service glance] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:54.005565 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-ced97c1e-0c54-4590-8888-d959c75f2994 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:54.090514 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ced97c1e-0c54-4590-8888-d959c75f2994 service glance] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:54.098247 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b1596f8-8440-4006-a4c1-84aa687a94a3 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a7f7afc3d44ec788c05d005e52a1b8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:54.098898 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b1596f8-8440-4006-a4c1-84aa687a94a3 service swift] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:54.099188 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b1596f8-8440-4006-a4c1-84aa687a94a3 service swift] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:54.099497 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b1596f8-8440-4006-a4c1-84aa687a94a3 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:54.110713 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b1596f8-8440-4006-a4c1-84aa687a94a3 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:54.112480 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b1596f8-8440-4006-a4c1-84aa687a94a3 service swift] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:54.131055 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1203/2372] 10.222.0.77 () {60 vars in 1384 bytes} [Tue Aug 30 05:13:54 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 127 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 30 05:13:54.673110 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a686a6-bb69-4f2a-8f02-5fd82d5d9004 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:54.673462 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a686a6-bb69-4f2a-8f02-5fd82d5d9004 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:54.673801 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a686a6-bb69-4f2a-8f02-5fd82d5d9004 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:54.694060 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-95a686a6-bb69-4f2a-8f02-5fd82d5d9004 None None] MFA Rules not processed for user `fe51dbec91624ed1ab00b441e29a3dd5`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:13:54.695457 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-95a686a6-bb69-4f2a-8f02-5fd82d5d9004 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:54.702858 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1170/2373] 10.222.0.77 () {58 vars in 967 bytes} [Tue Aug 30 05:13:54 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:13:54.716124 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b1596f8-8440-4006-a4c1-84aa687a94a3 service swift] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:54.727026 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b1596f8-8440-4006-a4c1-84aa687a94a3 service swift] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:54.734956 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f47f148-6547-46bd-b761-fc06cdd2c39f service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a7f7afc3d44ec788c05d005e52a1b8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:54.735759 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f47f148-6547-46bd-b761-fc06cdd2c39f service swift] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:54.736111 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f47f148-6547-46bd-b761-fc06cdd2c39f service swift] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:54.736446 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f47f148-6547-46bd-b761-fc06cdd2c39f service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:54.745620 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f47f148-6547-46bd-b761-fc06cdd2c39f service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:54.747163 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f47f148-6547-46bd-b761-fc06cdd2c39f service swift] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:54.755983 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1204/2374] 10.222.0.77 () {60 vars in 1384 bytes} [Tue Aug 30 05:13:54 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 30 05:13:55.013429 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da713ffb-7e74-4024-b624-8138b62c1428 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:55.013874 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da713ffb-7e74-4024-b624-8138b62c1428 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:55.013983 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da713ffb-7e74-4024-b624-8138b62c1428 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:55.034373 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-da713ffb-7e74-4024-b624-8138b62c1428 None None] MFA Rules not processed for user `fe51dbec91624ed1ab00b441e29a3dd5`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:13:55.035686 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-da713ffb-7e74-4024-b624-8138b62c1428 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:55.042621 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1171/2375] 10.222.0.77 () {58 vars in 1009 bytes} [Tue Aug 30 05:13:55 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:13:55.054956 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f47f148-6547-46bd-b761-fc06cdd2c39f service swift] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:55.066506 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f47f148-6547-46bd-b761-fc06cdd2c39f service swift] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:55.072013 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbbf24d3-08a5-4151-8cd6-5acb160948e3 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a7f7afc3d44ec788c05d005e52a1b8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:55.072606 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbbf24d3-08a5-4151-8cd6-5acb160948e3 service swift] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:55.072885 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbbf24d3-08a5-4151-8cd6-5acb160948e3 service swift] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:55.073157 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbbf24d3-08a5-4151-8cd6-5acb160948e3 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:55.081198 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbbf24d3-08a5-4151-8cd6-5acb160948e3 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:55.082771 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbbf24d3-08a5-4151-8cd6-5acb160948e3 service swift] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:55.087326 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:55.091376 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1205/2376] 10.222.0.77 () {60 vars in 1384 bytes} [Tue Aug 30 05:13:55 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 30 05:13:55.428797 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-097801bb-eb1e-4db6-a4c5-34932f935061 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:55.429185 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-097801bb-eb1e-4db6-a4c5-34932f935061 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:55.429465 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-097801bb-eb1e-4db6-a4c5-34932f935061 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:55.448626 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-097801bb-eb1e-4db6-a4c5-34932f935061 None None] MFA Rules not processed for user `fe51dbec91624ed1ab00b441e29a3dd5`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:13:55.449831 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-097801bb-eb1e-4db6-a4c5-34932f935061 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:55.458433 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1172/2377] 10.222.0.77 () {58 vars in 967 bytes} [Tue Aug 30 05:13:55 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:13:55.474537 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbbf24d3-08a5-4151-8cd6-5acb160948e3 service swift] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:55.487598 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbbf24d3-08a5-4151-8cd6-5acb160948e3 service swift] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:55.493271 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dba282a4-bf60-4ca1-b89a-8dde4ccacfb7 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a7f7afc3d44ec788c05d005e52a1b8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:55.493878 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba282a4-bf60-4ca1-b89a-8dde4ccacfb7 service swift] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:55.494157 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba282a4-bf60-4ca1-b89a-8dde4ccacfb7 service swift] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:55.494428 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba282a4-bf60-4ca1-b89a-8dde4ccacfb7 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:55.503056 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dba282a4-bf60-4ca1-b89a-8dde4ccacfb7 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:55.504664 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dba282a4-bf60-4ca1-b89a-8dde4ccacfb7 service swift] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:55.513130 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1206/2378] 10.222.0.77 () {60 vars in 1384 bytes} [Tue Aug 30 05:13:55 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 30 05:13:57.645323 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-097801bb-eb1e-4db6-a4c5-34932f935061 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:57.646835 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-097801bb-eb1e-4db6-a4c5-34932f935061 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:57.741335 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-097801bb-eb1e-4db6-a4c5-34932f935061 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:57.746637 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bc011fd-be4a-4967-9e46-1d6748070964 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:57.747413 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc011fd-be4a-4967-9e46-1d6748070964 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:57.747758 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc011fd-be4a-4967-9e46-1d6748070964 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:57.748118 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc011fd-be4a-4967-9e46-1d6748070964 admin admin] PATH_INFO: `/v3/users/71e9c605498544c8b6c75349d9b66a94` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:57.749501 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bc011fd-be4a-4967-9e46-1d6748070964 admin admin] RBAC: Authorizing `identity:delete_user(user_id=71e9c605498544c8b6c75349d9b66a94)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:57.751734 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bc011fd-be4a-4967-9e46-1d6748070964 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:57.822227 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-7bc011fd-be4a-4967-9e46-1d6748070964 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '71e9c605498544c8b6c75349d9b66a94'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:57.823030 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-7bc011fd-be4a-4967-9e46-1d6748070964 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '71e9c605498544c8b6c75349d9b66a94'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:57.823864 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-7bc011fd-be4a-4967-9e46-1d6748070964 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '71e9c605498544c8b6c75349d9b66a94'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:57.838900 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-7bc011fd-be4a-4967-9e46-1d6748070964 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '71e9c605498544c8b6c75349d9b66a94'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:57.847921 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-7bc011fd-be4a-4967-9e46-1d6748070964 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '71e9c605498544c8b6c75349d9b66a94'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:57.877222 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1173/2379] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:13:57 2022] DELETE /identity/v3/users/71e9c605498544c8b6c75349d9b66a94 => generated 0 bytes in 229 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:13:57.879485 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30cb4bfc-d0b8-49dc-ad0a-54bc3589e206 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:57.879957 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30cb4bfc-d0b8-49dc-ad0a-54bc3589e206 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:57.880075 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30cb4bfc-d0b8-49dc-ad0a-54bc3589e206 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:57.912763 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-30cb4bfc-d0b8-49dc-ad0a-54bc3589e206 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:13:57.914288 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-30cb4bfc-d0b8-49dc-ad0a-54bc3589e206 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:58.013788 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1207/2380] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:13:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:13:58.024867 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bc011fd-be4a-4967-9e46-1d6748070964 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:58.026320 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-7bc011fd-be4a-4967-9e46-1d6748070964 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:58.063141 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:13:58.098515 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bc011fd-be4a-4967-9e46-1d6748070964 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:58.102595 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00ee3714-8961-4980-9ddf-26f46eed3a30 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:58.103385 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00ee3714-8961-4980-9ddf-26f46eed3a30 service neutron] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:58.103690 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00ee3714-8961-4980-9ddf-26f46eed3a30 service neutron] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:58.103967 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00ee3714-8961-4980-9ddf-26f46eed3a30 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:58.118372 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00ee3714-8961-4980-9ddf-26f46eed3a30 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:58.120945 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00ee3714-8961-4980-9ddf-26f46eed3a30 service neutron] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:58.134534 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1174/2381] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:13:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 110 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:13:58.383728 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30cb4bfc-d0b8-49dc-ad0a-54bc3589e206 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:13:58.385155 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-30cb4bfc-d0b8-49dc-ad0a-54bc3589e206 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:13:58.397702 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30cb4bfc-d0b8-49dc-ad0a-54bc3589e206 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:13:58.403347 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25d5c24f-1ae7-4d12-a0b7-6811c70e8624 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:13:58.404088 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d5c24f-1ae7-4d12-a0b7-6811c70e8624 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:13:58.404616 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d5c24f-1ae7-4d12-a0b7-6811c70e8624 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:13:58.404977 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d5c24f-1ae7-4d12-a0b7-6811c70e8624 admin admin] PATH_INFO: `/v3/projects/5ba05d601a9240bb922f4d6dc3cc6bc6` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:13:58.406610 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25d5c24f-1ae7-4d12-a0b7-6811c70e8624 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5ba05d601a9240bb922f4d6dc3cc6bc6)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:13:58.408766 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25d5c24f-1ae7-4d12-a0b7-6811c70e8624 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:13:58.470944 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-25d5c24f-1ae7-4d12-a0b7-6811c70e8624 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5ba05d601a9240bb922f4d6dc3cc6bc6', 'request_id': 'req-25d5c24f-1ae7-4d12-a0b7-6811c70e8624'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:58.479304 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-25d5c24f-1ae7-4d12-a0b7-6811c70e8624 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5ba05d601a9240bb922f4d6dc3cc6bc6', 'request_id': 'req-25d5c24f-1ae7-4d12-a0b7-6811c70e8624'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:58.486599 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-25d5c24f-1ae7-4d12-a0b7-6811c70e8624 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5ba05d601a9240bb922f4d6dc3cc6bc6', 'request_id': 'req-25d5c24f-1ae7-4d12-a0b7-6811c70e8624'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:58.509057 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-25d5c24f-1ae7-4d12-a0b7-6811c70e8624 admin admin] The token cache is being invalidate because project 5ba05d601a9240bb922f4d6dc3cc6bc6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81625) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:13:58.509351 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-25d5c24f-1ae7-4d12-a0b7-6811c70e8624 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:58.510016 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-25d5c24f-1ae7-4d12-a0b7-6811c70e8624 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:13:58.511566 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1208/2382] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:13:58 2022] DELETE /identity/v3/projects/5ba05d601a9240bb922f4d6dc3cc6bc6 => generated 0 bytes in 128 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:14:04.015619 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef39b2d9-494c-4dab-b9b5-799751c13bc3 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:14:04.016287 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef39b2d9-494c-4dab-b9b5-799751c13bc3 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:14:04.016474 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef39b2d9-494c-4dab-b9b5-799751c13bc3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:14:04.038120 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-ef39b2d9-494c-4dab-b9b5-799751c13bc3 None None] MFA Rules not processed for user `fe51dbec91624ed1ab00b441e29a3dd5`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:14:04.039468 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-ef39b2d9-494c-4dab-b9b5-799751c13bc3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:14:04.100246 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1175/2383] 10.222.0.77 () {58 vars in 1009 bytes} [Tue Aug 30 05:14:04 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 05:14:04.112405 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25d5c24f-1ae7-4d12-a0b7-6811c70e8624 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:14:04.113825 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-25d5c24f-1ae7-4d12-a0b7-6811c70e8624 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:14:04.186067 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25d5c24f-1ae7-4d12-a0b7-6811c70e8624 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:14:04.191154 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91404e35-9d38-46d7-bade-745017627a7d service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a7f7afc3d44ec788c05d005e52a1b8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:14:04.191748 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91404e35-9d38-46d7-bade-745017627a7d service swift] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:14:04.192026 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91404e35-9d38-46d7-bade-745017627a7d service swift] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:14:04.192293 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91404e35-9d38-46d7-bade-745017627a7d service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:14:04.201317 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91404e35-9d38-46d7-bade-745017627a7d service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:14:04.202928 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91404e35-9d38-46d7-bade-745017627a7d service swift] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:14:04.208222 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:14:04.212512 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1209/2384] 10.222.0.77 () {60 vars in 1384 bytes} [Tue Aug 30 05:14:04 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 30 05:14:11.445786 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef39b2d9-494c-4dab-b9b5-799751c13bc3 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:14:11.447615 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-ef39b2d9-494c-4dab-b9b5-799751c13bc3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:14:11.529254 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef39b2d9-494c-4dab-b9b5-799751c13bc3 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:14:11.535102 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51007de1-c05a-4736-bcd8-ba3bd272cfd0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:14:11.535918 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51007de1-c05a-4736-bcd8-ba3bd272cfd0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:14:11.536360 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51007de1-c05a-4736-bcd8-ba3bd272cfd0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:14:11.536782 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51007de1-c05a-4736-bcd8-ba3bd272cfd0 admin admin] PATH_INFO: `/v3/users/3274472d6c9944b395d11d7aa68c664a` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:14:11.538111 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51007de1-c05a-4736-bcd8-ba3bd272cfd0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3274472d6c9944b395d11d7aa68c664a)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:14:11.540356 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51007de1-c05a-4736-bcd8-ba3bd272cfd0 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:14:11.597077 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-51007de1-c05a-4736-bcd8-ba3bd272cfd0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3274472d6c9944b395d11d7aa68c664a'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:14:11.597725 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-51007de1-c05a-4736-bcd8-ba3bd272cfd0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3274472d6c9944b395d11d7aa68c664a'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:14:11.598558 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-51007de1-c05a-4736-bcd8-ba3bd272cfd0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3274472d6c9944b395d11d7aa68c664a'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:14:11.611103 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-51007de1-c05a-4736-bcd8-ba3bd272cfd0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3274472d6c9944b395d11d7aa68c664a'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:14:11.620343 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-51007de1-c05a-4736-bcd8-ba3bd272cfd0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3274472d6c9944b395d11d7aa68c664a'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:14:11.623242 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=91,keystone:DELETE=9 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:14:11.636161 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1176/2385] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:14:11 2022] DELETE /identity/v3/users/3274472d6c9944b395d11d7aa68c664a => generated 0 bytes in 191 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:14:11.641536 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26d7a560-dee8-499a-b264-177517f099cc None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:14:11.642074 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26d7a560-dee8-499a-b264-177517f099cc None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:14:11.642074 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26d7a560-dee8-499a-b264-177517f099cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:14:11.666890 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-26d7a560-dee8-499a-b264-177517f099cc None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:14:11.668448 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-26d7a560-dee8-499a-b264-177517f099cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:14:11.737775 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1210/2386] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:14:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:14:11.748020 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51007de1-c05a-4736-bcd8-ba3bd272cfd0 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:14:11.749436 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-51007de1-c05a-4736-bcd8-ba3bd272cfd0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:14:11.809302 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51007de1-c05a-4736-bcd8-ba3bd272cfd0 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:14:11.813140 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1beb48cd-c010-4f7e-b880-22ef91a1c118 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:14:11.813822 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1beb48cd-c010-4f7e-b880-22ef91a1c118 service neutron] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:14:11.814163 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1beb48cd-c010-4f7e-b880-22ef91a1c118 service neutron] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:14:11.814440 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1beb48cd-c010-4f7e-b880-22ef91a1c118 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:14:11.824716 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1beb48cd-c010-4f7e-b880-22ef91a1c118 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:14:11.826327 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1beb48cd-c010-4f7e-b880-22ef91a1c118 service neutron] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:14:11.838341 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1177/2387] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:14:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:14:12.203974 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26d7a560-dee8-499a-b264-177517f099cc None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:14:12.205989 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-26d7a560-dee8-499a-b264-177517f099cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:14:12.245328 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26d7a560-dee8-499a-b264-177517f099cc None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:14:12.251119 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7bb9d6c-20cd-46f6-9115-604794217b7c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:14:12.252015 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7bb9d6c-20cd-46f6-9115-604794217b7c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:14:12.252916 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7bb9d6c-20cd-46f6-9115-604794217b7c admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:14:12.253734 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7bb9d6c-20cd-46f6-9115-604794217b7c admin admin] PATH_INFO: `/v3/projects/415a364df08041ed9fa305477fadf421` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:14:12.254944 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7bb9d6c-20cd-46f6-9115-604794217b7c admin admin] RBAC: Authorizing `identity:delete_project(project_id=415a364df08041ed9fa305477fadf421)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:14:12.256968 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7bb9d6c-20cd-46f6-9115-604794217b7c admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:14:12.316133 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-e7bb9d6c-20cd-46f6-9115-604794217b7c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '415a364df08041ed9fa305477fadf421', 'request_id': 'req-e7bb9d6c-20cd-46f6-9115-604794217b7c'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:14:12.333616 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-e7bb9d6c-20cd-46f6-9115-604794217b7c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '415a364df08041ed9fa305477fadf421', 'request_id': 'req-e7bb9d6c-20cd-46f6-9115-604794217b7c'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:14:12.341915 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-e7bb9d6c-20cd-46f6-9115-604794217b7c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '415a364df08041ed9fa305477fadf421', 'request_id': 'req-e7bb9d6c-20cd-46f6-9115-604794217b7c'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:14:12.358707 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-e7bb9d6c-20cd-46f6-9115-604794217b7c admin admin] The token cache is being invalidate because project 415a364df08041ed9fa305477fadf421 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81625) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:14:12.359095 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-e7bb9d6c-20cd-46f6-9115-604794217b7c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:14:12.360483 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-e7bb9d6c-20cd-46f6-9115-604794217b7c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:14:12.362556 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1211/2388] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:14:12 2022] DELETE /identity/v3/projects/415a364df08041ed9fa305477fadf421 => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:14:12.371503 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7b0452b-5161-4f4c-8819-31cef986ebb0 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:14:12.371503 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7b0452b-5161-4f4c-8819-31cef986ebb0 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:14:12.371503 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7b0452b-5161-4f4c-8819-31cef986ebb0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:14:12.395989 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-e7b0452b-5161-4f4c-8819-31cef986ebb0 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:14:12.397210 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-e7b0452b-5161-4f4c-8819-31cef986ebb0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:14:12.507848 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1178/2389] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:14:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:14:12.516911 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a69b54d6-575f-4be9-8ece-3a06f1290404 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:14:12.517194 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a69b54d6-575f-4be9-8ece-3a06f1290404 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:14:12.517524 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a69b54d6-575f-4be9-8ece-3a06f1290404 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:14:12.541944 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-a69b54d6-575f-4be9-8ece-3a06f1290404 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:14:12.543346 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-a69b54d6-575f-4be9-8ece-3a06f1290404 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:14:12.551866 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1212/2390] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:14:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:14:12.556251 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7b0452b-5161-4f4c-8819-31cef986ebb0 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:14:12.566725 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7b0452b-5161-4f4c-8819-31cef986ebb0 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:14:12.571677 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60972e11-d707-4a21-9823-ab3d0ca0c6a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:14:12.572581 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60972e11-d707-4a21-9823-ab3d0ca0c6a4 admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:14:12.572863 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60972e11-d707-4a21-9823-ab3d0ca0c6a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:14:12.573131 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60972e11-d707-4a21-9823-ab3d0ca0c6a4 admin admin] PATH_INFO: `/v3/domains` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:14:12.573782 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60972e11-d707-4a21-9823-ab3d0ca0c6a4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:14:12.574195 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60972e11-d707-4a21-9823-ab3d0ca0c6a4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:14:12.576018 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60972e11-d707-4a21-9823-ab3d0ca0c6a4 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:14:12.590650 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1179/2391] 10.222.0.77 () {58 vars in 1153 bytes} [Tue Aug 30 05:14:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 05:14:12.595057 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a69b54d6-575f-4be9-8ece-3a06f1290404 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:14:12.605788 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a69b54d6-575f-4be9-8ece-3a06f1290404 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:14:12.611850 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc65055-a6b7-447f-b403-390d16ba3737 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:14:12.612983 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc65055-a6b7-447f-b403-390d16ba3737 admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:14:12.613811 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc65055-a6b7-447f-b403-390d16ba3737 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:14:12.614238 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc65055-a6b7-447f-b403-390d16ba3737 admin admin] PATH_INFO: `/v3/projects` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:14:12.615422 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cc65055-a6b7-447f-b403-390d16ba3737 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:14:12.618155 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cc65055-a6b7-447f-b403-390d16ba3737 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:14:12.667731 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1213/2392] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:14:12 2022] POST /identity/v3/projects => generated 371 bytes in 73 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 05:14:12.671881 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60972e11-d707-4a21-9823-ab3d0ca0c6a4 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:14:12.683928 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60972e11-d707-4a21-9823-ab3d0ca0c6a4 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:14:12.776166 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc01a85-3fbb-4b88-8825-7058d51cb9e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:14:12.777222 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc01a85-3fbb-4b88-8825-7058d51cb9e9 admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:14:12.777510 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc01a85-3fbb-4b88-8825-7058d51cb9e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:14:12.780200 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc01a85-3fbb-4b88-8825-7058d51cb9e9 admin admin] PATH_INFO: `/v3/users` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:14:12.780200 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cc01a85-3fbb-4b88-8825-7058d51cb9e9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:14:12.781458 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cc01a85-3fbb-4b88-8825-7058d51cb9e9 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:14:12.806040 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1180/2393] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:14:12 2022] POST /identity/v3/users => generated 286 bytes in 135 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:14:12.809927 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc65055-a6b7-447f-b403-390d16ba3737 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:14:12.821166 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc65055-a6b7-447f-b403-390d16ba3737 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:14:12.826098 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b08b126d-8f02-4b04-acd1-71e0856a0f56 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:14:12.827001 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b08b126d-8f02-4b04-acd1-71e0856a0f56 admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:14:12.827341 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b08b126d-8f02-4b04-acd1-71e0856a0f56 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:14:12.827679 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b08b126d-8f02-4b04-acd1-71e0856a0f56 admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:14:12.828387 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b08b126d-8f02-4b04-acd1-71e0856a0f56 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:14:12.830043 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b08b126d-8f02-4b04-acd1-71e0856a0f56 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:14:12.841248 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1214/2394] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:14:12 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 05:14:12.845510 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc01a85-3fbb-4b88-8825-7058d51cb9e9 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:14:12.855957 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc01a85-3fbb-4b88-8825-7058d51cb9e9 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:14:12.862072 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aaa98c6-0b75-4d0e-916b-bfdf9c1c8b41 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:14:12.862704 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aaa98c6-0b75-4d0e-916b-bfdf9c1c8b41 admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:14:12.863028 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aaa98c6-0b75-4d0e-916b-bfdf9c1c8b41 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:14:12.863301 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aaa98c6-0b75-4d0e-916b-bfdf9c1c8b41 admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:14:12.863944 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aaa98c6-0b75-4d0e-916b-bfdf9c1c8b41 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:14:12.865653 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aaa98c6-0b75-4d0e-916b-bfdf9c1c8b41 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:14:12.875077 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:INSERT=4,keystone:DELETE=1,keystone:SELECT=95 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:14:12.888881 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1181/2395] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:14:12 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:14:12.890362 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b08b126d-8f02-4b04-acd1-71e0856a0f56 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:14:12.901983 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b08b126d-8f02-4b04-acd1-71e0856a0f56 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:14:12.907005 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6033440a-b8cf-4988-9f9f-434e221c8294 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:14:12.907656 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6033440a-b8cf-4988-9f9f-434e221c8294 admin admin] REQUEST_METHOD: `PUT` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:14:12.907935 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6033440a-b8cf-4988-9f9f-434e221c8294 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:14:12.908211 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6033440a-b8cf-4988-9f9f-434e221c8294 admin admin] PATH_INFO: `/v3/projects/5709c7e42ba84cf797ea1b4bb105c1dc/users/b676c77fd6554381a0d91866da23f500/roles/6f03d86980c34c36b44a8d6764ff4fda` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:14:12.927829 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6033440a-b8cf-4988-9f9f-434e221c8294 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5709c7e42ba84cf797ea1b4bb105c1dc, user_id=b676c77fd6554381a0d91866da23f500, role_id=6f03d86980c34c36b44a8d6764ff4fda)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:14:12.930257 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6033440a-b8cf-4988-9f9f-434e221c8294 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:14:12.946033 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1215/2396] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:14:12 2022] PUT /identity/v3/projects/5709c7e42ba84cf797ea1b4bb105c1dc/users/b676c77fd6554381a0d91866da23f500/roles/6f03d86980c34c36b44a8d6764ff4fda => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:14:12.956179 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad21ac7b-3cfa-4935-8a18-8e67a527287d None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:14:12.956459 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad21ac7b-3cfa-4935-8a18-8e67a527287d None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:14:12.956731 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad21ac7b-3cfa-4935-8a18-8e67a527287d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:14:12.987770 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-ad21ac7b-3cfa-4935-8a18-8e67a527287d None None] MFA Rules not processed for user `b676c77fd6554381a0d91866da23f500`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:14:12.988977 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-ad21ac7b-3cfa-4935-8a18-8e67a527287d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:14:13.068490 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1182/2397] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:14:12 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:14:13.079523 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6033440a-b8cf-4988-9f9f-434e221c8294 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:14:13.081296 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-6033440a-b8cf-4988-9f9f-434e221c8294 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:14:13.108679 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:14:13.150064 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6033440a-b8cf-4988-9f9f-434e221c8294 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:14:13.159619 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-921d1f80-ec9e-43e1-9be4-d336a4f2cdad service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '019a79b2affe478b8a30d1c538d0415c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:14:13.160239 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-921d1f80-ec9e-43e1-9be4-d336a4f2cdad service cinder] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:14:13.160517 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-921d1f80-ec9e-43e1-9be4-d336a4f2cdad service cinder] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:14:13.160791 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-921d1f80-ec9e-43e1-9be4-d336a4f2cdad service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:14:13.170527 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-921d1f80-ec9e-43e1-9be4-d336a4f2cdad service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:14:13.172009 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-921d1f80-ec9e-43e1-9be4-d336a4f2cdad service cinder] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:14:13.183262 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1216/2398] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:14:13 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:14:23.065086 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=32 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:14:23.177903 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=17 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:14:24.082524 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad21ac7b-3cfa-4935-8a18-8e67a527287d None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:14:24.220501 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad21ac7b-3cfa-4935-8a18-8e67a527287d None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:14:24.228831 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f20c06a-06b1-4a05-a7f0-da59d913a095 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:14:24.229801 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f20c06a-06b1-4a05-a7f0-da59d913a095 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:14:24.230284 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f20c06a-06b1-4a05-a7f0-da59d913a095 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:14:24.230765 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f20c06a-06b1-4a05-a7f0-da59d913a095 admin admin] PATH_INFO: `/v3/users/b676c77fd6554381a0d91866da23f500` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:14:24.232194 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f20c06a-06b1-4a05-a7f0-da59d913a095 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b676c77fd6554381a0d91866da23f500)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:14:24.234367 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f20c06a-06b1-4a05-a7f0-da59d913a095 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:14:24.329170 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-7f20c06a-06b1-4a05-a7f0-da59d913a095 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b676c77fd6554381a0d91866da23f500'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:14:24.331133 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-7f20c06a-06b1-4a05-a7f0-da59d913a095 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b676c77fd6554381a0d91866da23f500'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:14:24.332279 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-7f20c06a-06b1-4a05-a7f0-da59d913a095 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b676c77fd6554381a0d91866da23f500'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:14:24.366760 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-7f20c06a-06b1-4a05-a7f0-da59d913a095 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b676c77fd6554381a0d91866da23f500'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:14:24.379357 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-7f20c06a-06b1-4a05-a7f0-da59d913a095 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b676c77fd6554381a0d91866da23f500'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:14:24.413839 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1183/2399] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:14:24 2022] DELETE /identity/v3/users/b676c77fd6554381a0d91866da23f500 => generated 0 bytes in 332 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:14:24.420202 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d41e6170-c171-44c7-84b8-1c0332e01f78 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:14:24.421047 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d41e6170-c171-44c7-84b8-1c0332e01f78 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:14:24.421047 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d41e6170-c171-44c7-84b8-1c0332e01f78 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:14:24.473221 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-d41e6170-c171-44c7-84b8-1c0332e01f78 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:14:24.475951 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-d41e6170-c171-44c7-84b8-1c0332e01f78 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:14:24.606962 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1217/2400] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:14:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:14:24.619536 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f20c06a-06b1-4a05-a7f0-da59d913a095 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:14:24.621139 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-7f20c06a-06b1-4a05-a7f0-da59d913a095 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:14:24.710141 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f20c06a-06b1-4a05-a7f0-da59d913a095 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:14:24.714151 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40a69fbf-e977-4a10-b6da-eca20d4e8448 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:14:24.714935 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a69fbf-e977-4a10-b6da-eca20d4e8448 service neutron] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:14:24.715283 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a69fbf-e977-4a10-b6da-eca20d4e8448 service neutron] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:14:24.715629 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a69fbf-e977-4a10-b6da-eca20d4e8448 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:14:24.726866 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40a69fbf-e977-4a10-b6da-eca20d4e8448 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:14:24.728325 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40a69fbf-e977-4a10-b6da-eca20d4e8448 service neutron] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:14:24.742347 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1184/2401] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:14:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 123 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 05:14:25.174228 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d41e6170-c171-44c7-84b8-1c0332e01f78 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:14:25.177189 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-d41e6170-c171-44c7-84b8-1c0332e01f78 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:14:25.197307 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d41e6170-c171-44c7-84b8-1c0332e01f78 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:14:25.202200 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fe242a2-4fb8-4cee-a8dc-bf947917d2f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:14:25.202941 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fe242a2-4fb8-4cee-a8dc-bf947917d2f9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:14:25.203440 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fe242a2-4fb8-4cee-a8dc-bf947917d2f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:14:25.203816 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fe242a2-4fb8-4cee-a8dc-bf947917d2f9 admin admin] PATH_INFO: `/v3/projects/5709c7e42ba84cf797ea1b4bb105c1dc` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:14:25.205052 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fe242a2-4fb8-4cee-a8dc-bf947917d2f9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5709c7e42ba84cf797ea1b4bb105c1dc)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:14:25.207253 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fe242a2-4fb8-4cee-a8dc-bf947917d2f9 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:14:25.267933 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-7fe242a2-4fb8-4cee-a8dc-bf947917d2f9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5709c7e42ba84cf797ea1b4bb105c1dc', 'request_id': 'req-7fe242a2-4fb8-4cee-a8dc-bf947917d2f9'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:14:25.278559 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-7fe242a2-4fb8-4cee-a8dc-bf947917d2f9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5709c7e42ba84cf797ea1b4bb105c1dc', 'request_id': 'req-7fe242a2-4fb8-4cee-a8dc-bf947917d2f9'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:14:25.289400 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-7fe242a2-4fb8-4cee-a8dc-bf947917d2f9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5709c7e42ba84cf797ea1b4bb105c1dc', 'request_id': 'req-7fe242a2-4fb8-4cee-a8dc-bf947917d2f9'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:14:25.302703 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-7fe242a2-4fb8-4cee-a8dc-bf947917d2f9 admin admin] The token cache is being invalidate because project 5709c7e42ba84cf797ea1b4bb105c1dc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81625) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:14:25.303311 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-7fe242a2-4fb8-4cee-a8dc-bf947917d2f9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:14:25.304069 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-7fe242a2-4fb8-4cee-a8dc-bf947917d2f9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:14:25.305874 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1218/2402] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:14:25 2022] DELETE /identity/v3/projects/5709c7e42ba84cf797ea1b4bb105c1dc => generated 0 bytes in 132 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:14:25.312406 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5775253a-9662-4c5d-a21f-0c4a133e36ec None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:14:25.313079 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5775253a-9662-4c5d-a21f-0c4a133e36ec None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:14:25.313524 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5775253a-9662-4c5d-a21f-0c4a133e36ec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:14:25.344935 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-5775253a-9662-4c5d-a21f-0c4a133e36ec None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:14:25.346840 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-5775253a-9662-4c5d-a21f-0c4a133e36ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:14:25.372192 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:14:25.450625 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-075df959-cfd7-4e03-8e63-d679b5571370 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:14:25.451592 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-075df959-cfd7-4e03-8e63-d679b5571370 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:14:25.452046 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-075df959-cfd7-4e03-8e63-d679b5571370 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:14:25.475236 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1185/2403] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:14:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:14:25.517683 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-075df959-cfd7-4e03-8e63-d679b5571370 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:14:25.520131 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-075df959-cfd7-4e03-8e63-d679b5571370 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:14:25.534183 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1219/2404] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:14:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:14:25.539344 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5775253a-9662-4c5d-a21f-0c4a133e36ec None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:14:25.554627 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5775253a-9662-4c5d-a21f-0c4a133e36ec None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:14:25.559825 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6fe1b04-c34e-4d93-9c3e-d15a2d5b891b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:14:25.560524 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6fe1b04-c34e-4d93-9c3e-d15a2d5b891b admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:14:25.560864 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6fe1b04-c34e-4d93-9c3e-d15a2d5b891b admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:14:25.561199 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6fe1b04-c34e-4d93-9c3e-d15a2d5b891b admin admin] PATH_INFO: `/v3/domains` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:14:25.561947 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6fe1b04-c34e-4d93-9c3e-d15a2d5b891b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:14:25.562309 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6fe1b04-c34e-4d93-9c3e-d15a2d5b891b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:14:25.565661 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6fe1b04-c34e-4d93-9c3e-d15a2d5b891b admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:14:25.584530 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1186/2405] 10.222.0.77 () {58 vars in 1153 bytes} [Tue Aug 30 05:14:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:14:25.588673 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-075df959-cfd7-4e03-8e63-d679b5571370 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:14:25.599370 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-075df959-cfd7-4e03-8e63-d679b5571370 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:14:25.612646 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67d39ddf-b5d5-4b76-9ae0-44ee40bbff8f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:14:25.613202 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67d39ddf-b5d5-4b76-9ae0-44ee40bbff8f admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:14:25.613480 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67d39ddf-b5d5-4b76-9ae0-44ee40bbff8f admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:14:25.613747 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67d39ddf-b5d5-4b76-9ae0-44ee40bbff8f admin admin] PATH_INFO: `/v3/projects` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:14:25.614999 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67d39ddf-b5d5-4b76-9ae0-44ee40bbff8f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:14:25.617101 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67d39ddf-b5d5-4b76-9ae0-44ee40bbff8f admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:14:25.655364 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6fe1b04-c34e-4d93-9c3e-d15a2d5b891b admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:14:25.659499 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1220/2406] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:14:25 2022] POST /identity/v3/projects => generated 391 bytes in 71 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:14:25.679585 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6fe1b04-c34e-4d93-9c3e-d15a2d5b891b admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:14:25.794906 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f36c45de-b55f-43a7-a6ed-743f7ecdbc92 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:14:25.795545 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f36c45de-b55f-43a7-a6ed-743f7ecdbc92 admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:14:25.795892 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f36c45de-b55f-43a7-a6ed-743f7ecdbc92 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:14:25.796090 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f36c45de-b55f-43a7-a6ed-743f7ecdbc92 admin admin] PATH_INFO: `/v3/users` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:14:25.796718 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f36c45de-b55f-43a7-a6ed-743f7ecdbc92 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:14:25.798874 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f36c45de-b55f-43a7-a6ed-743f7ecdbc92 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:14:25.830869 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1187/2407] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:14:25 2022] POST /identity/v3/users => generated 296 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:14:25.832231 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67d39ddf-b5d5-4b76-9ae0-44ee40bbff8f admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:14:25.846150 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67d39ddf-b5d5-4b76-9ae0-44ee40bbff8f admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:14:25.856649 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89b53b5b-44a5-48bd-aaee-a8788c6ee57d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:14:25.857304 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b53b5b-44a5-48bd-aaee-a8788c6ee57d admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:14:25.857657 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b53b5b-44a5-48bd-aaee-a8788c6ee57d admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:14:25.857877 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b53b5b-44a5-48bd-aaee-a8788c6ee57d admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:14:25.858521 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89b53b5b-44a5-48bd-aaee-a8788c6ee57d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:14:25.860284 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89b53b5b-44a5-48bd-aaee-a8788c6ee57d admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:14:25.875391 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1221/2408] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:14:25 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 05:14:25.882867 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f36c45de-b55f-43a7-a6ed-743f7ecdbc92 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:14:25.897271 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f36c45de-b55f-43a7-a6ed-743f7ecdbc92 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:14:25.919455 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-874fcb88-8fdc-477f-bd5f-7094f05b6773 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:14:25.920492 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-874fcb88-8fdc-477f-bd5f-7094f05b6773 admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:14:25.920776 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-874fcb88-8fdc-477f-bd5f-7094f05b6773 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:14:25.921415 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-874fcb88-8fdc-477f-bd5f-7094f05b6773 admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:14:25.922248 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-874fcb88-8fdc-477f-bd5f-7094f05b6773 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:14:25.924519 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-874fcb88-8fdc-477f-bd5f-7094f05b6773 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:14:25.944127 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1188/2409] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:14:25 2022] GET /identity/v3/roles => generated 1420 bytes in 65 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 05:14:25.948231 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89b53b5b-44a5-48bd-aaee-a8788c6ee57d admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:14:25.959733 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89b53b5b-44a5-48bd-aaee-a8788c6ee57d admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:14:25.964696 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ce49798-7d5c-4a4c-a43b-e05b91cd209c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:14:25.965355 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ce49798-7d5c-4a4c-a43b-e05b91cd209c admin admin] REQUEST_METHOD: `PUT` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:14:25.965632 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ce49798-7d5c-4a4c-a43b-e05b91cd209c admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:14:25.965900 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ce49798-7d5c-4a4c-a43b-e05b91cd209c admin admin] PATH_INFO: `/v3/projects/880e949296cb40018a76ad37b278e324/users/2627cbebc4c040f4adba8d6f97cace78/roles/6f03d86980c34c36b44a8d6764ff4fda` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:14:25.985952 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ce49798-7d5c-4a4c-a43b-e05b91cd209c admin admin] RBAC: Authorizing `identity:create_grant(project_id=880e949296cb40018a76ad37b278e324, user_id=2627cbebc4c040f4adba8d6f97cace78, role_id=6f03d86980c34c36b44a8d6764ff4fda)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:14:25.989009 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ce49798-7d5c-4a4c-a43b-e05b91cd209c admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:14:26.007345 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1222/2410] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:14:25 2022] PUT /identity/v3/projects/880e949296cb40018a76ad37b278e324/users/2627cbebc4c040f4adba8d6f97cace78/roles/6f03d86980c34c36b44a8d6764ff4fda => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:14:26.018585 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ac3f737-9477-459a-a61b-4f148f501037 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:14:26.018872 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ac3f737-9477-459a-a61b-4f148f501037 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:14:26.019324 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ac3f737-9477-459a-a61b-4f148f501037 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:14:26.055385 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-8ac3f737-9477-459a-a61b-4f148f501037 None None] MFA Rules not processed for user `2627cbebc4c040f4adba8d6f97cace78`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:14:26.057254 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-8ac3f737-9477-459a-a61b-4f148f501037 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:14:26.177494 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1189/2411] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:14:26 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:14:26.185054 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ce49798-7d5c-4a4c-a43b-e05b91cd209c admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:14:26.187257 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-0ce49798-7d5c-4a4c-a43b-e05b91cd209c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:14:26.239207 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:14:26.288453 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ce49798-7d5c-4a4c-a43b-e05b91cd209c admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:14:26.294331 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1014e950-4b51-4c27-8a02-627297aa8d43 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '019a79b2affe478b8a30d1c538d0415c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:14:26.294968 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1014e950-4b51-4c27-8a02-627297aa8d43 service cinder] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:14:26.295245 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1014e950-4b51-4c27-8a02-627297aa8d43 service cinder] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:14:26.295513 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1014e950-4b51-4c27-8a02-627297aa8d43 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:14:26.312444 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1014e950-4b51-4c27-8a02-627297aa8d43 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:14:26.314645 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1014e950-4b51-4c27-8a02-627297aa8d43 service cinder] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:14:26.337421 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1223/2412] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:14:26 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 153 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:14:33.061217 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bf930b2-52b8-4386-bcd5-62680dc9d1e3 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:14:33.061700 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bf930b2-52b8-4386-bcd5-62680dc9d1e3 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:14:33.061883 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bf930b2-52b8-4386-bcd5-62680dc9d1e3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:14:33.080925 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-3bf930b2-52b8-4386-bcd5-62680dc9d1e3 None None] MFA Rules not processed for user `fe51dbec91624ed1ab00b441e29a3dd5`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:14:33.082095 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-3bf930b2-52b8-4386-bcd5-62680dc9d1e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:14:33.094044 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:14:33.136738 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1190/2413] 10.222.0.77 () {58 vars in 967 bytes} [Tue Aug 30 05:14:33 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 76 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:14:33.149378 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1014e950-4b51-4c27-8a02-627297aa8d43 service cinder] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:14:33.150345 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-1014e950-4b51-4c27-8a02-627297aa8d43 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:14:33.219119 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1014e950-4b51-4c27-8a02-627297aa8d43 service cinder] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:14:33.224016 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-480a0526-48a2-4cef-b557-b1130fa1ae5c service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a7f7afc3d44ec788c05d005e52a1b8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:14:33.224603 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-480a0526-48a2-4cef-b557-b1130fa1ae5c service swift] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:14:33.224879 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-480a0526-48a2-4cef-b557-b1130fa1ae5c service swift] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:14:33.225172 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-480a0526-48a2-4cef-b557-b1130fa1ae5c service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:14:33.232955 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-480a0526-48a2-4cef-b557-b1130fa1ae5c service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:14:33.234566 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-480a0526-48a2-4cef-b557-b1130fa1ae5c service swift] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:14:33.242985 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1224/2414] 10.222.0.77 () {60 vars in 1384 bytes} [Tue Aug 30 05:14:33 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 30 05:14:36.458402 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9b2e253-1067-491f-b19a-99f9d94dedd1 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:14:36.459209 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9b2e253-1067-491f-b19a-99f9d94dedd1 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:14:36.459209 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9b2e253-1067-491f-b19a-99f9d94dedd1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:14:36.493078 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-f9b2e253-1067-491f-b19a-99f9d94dedd1 None None] MFA Rules not processed for user `fe51dbec91624ed1ab00b441e29a3dd5`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:14:36.494342 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-f9b2e253-1067-491f-b19a-99f9d94dedd1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:14:36.502385 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1191/2415] 10.222.0.77 () {58 vars in 1009 bytes} [Tue Aug 30 05:14:36 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 45 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:14:36.517018 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-480a0526-48a2-4cef-b557-b1130fa1ae5c service swift] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:14:36.528856 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-480a0526-48a2-4cef-b557-b1130fa1ae5c service swift] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:14:36.541503 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35355aa0-a874-4668-9ced-b90440ce1784 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a7f7afc3d44ec788c05d005e52a1b8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:14:36.542167 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35355aa0-a874-4668-9ced-b90440ce1784 service swift] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:14:36.542474 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35355aa0-a874-4668-9ced-b90440ce1784 service swift] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:14:36.543055 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35355aa0-a874-4668-9ced-b90440ce1784 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:14:36.554698 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35355aa0-a874-4668-9ced-b90440ce1784 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:14:36.556335 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35355aa0-a874-4668-9ced-b90440ce1784 service swift] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:14:36.565167 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1225/2416] 10.222.0.77 () {60 vars in 1384 bytes} [Tue Aug 30 05:14:36 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 49 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 30 05:14:37.444326 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9b2e253-1067-491f-b19a-99f9d94dedd1 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:14:37.446147 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-f9b2e253-1067-491f-b19a-99f9d94dedd1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:14:37.541834 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9b2e253-1067-491f-b19a-99f9d94dedd1 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:14:37.547149 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37e3fae0-052d-4edc-89b2-7aa45f8a51d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:14:37.547952 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37e3fae0-052d-4edc-89b2-7aa45f8a51d7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:14:37.548257 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37e3fae0-052d-4edc-89b2-7aa45f8a51d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:14:37.548761 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37e3fae0-052d-4edc-89b2-7aa45f8a51d7 admin admin] PATH_INFO: `/v3/users/b50d70f00f33494e81d4aac49df96dc4` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:14:37.550565 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37e3fae0-052d-4edc-89b2-7aa45f8a51d7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b50d70f00f33494e81d4aac49df96dc4)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:14:37.552699 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37e3fae0-052d-4edc-89b2-7aa45f8a51d7 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:14:37.638940 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-37e3fae0-052d-4edc-89b2-7aa45f8a51d7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b50d70f00f33494e81d4aac49df96dc4'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:14:37.640056 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-37e3fae0-052d-4edc-89b2-7aa45f8a51d7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b50d70f00f33494e81d4aac49df96dc4'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:14:37.641416 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-37e3fae0-052d-4edc-89b2-7aa45f8a51d7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b50d70f00f33494e81d4aac49df96dc4'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:14:37.659880 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-37e3fae0-052d-4edc-89b2-7aa45f8a51d7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b50d70f00f33494e81d4aac49df96dc4'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:14:37.675843 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-37e3fae0-052d-4edc-89b2-7aa45f8a51d7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b50d70f00f33494e81d4aac49df96dc4'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:14:37.693555 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1192/2417] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:14:37 2022] DELETE /identity/v3/users/b50d70f00f33494e81d4aac49df96dc4 => generated 0 bytes in 251 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:14:37.976724 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35355aa0-a874-4668-9ced-b90440ce1784 service swift] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:14:38.005574 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35355aa0-a874-4668-9ced-b90440ce1784 service swift] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:14:38.112134 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b1410ef-c33f-4023-80db-3810af5ee365 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:14:38.113285 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b1410ef-c33f-4023-80db-3810af5ee365 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:14:38.117365 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b1410ef-c33f-4023-80db-3810af5ee365 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:14:38.117928 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b1410ef-c33f-4023-80db-3810af5ee365 admin admin] PATH_INFO: `/v3/projects/9cea756f95e843b5868e53f6446d648a` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:14:38.120254 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b1410ef-c33f-4023-80db-3810af5ee365 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9cea756f95e843b5868e53f6446d648a)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:14:38.122182 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b1410ef-c33f-4023-80db-3810af5ee365 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:14:38.199589 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:14:38.210701 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-2b1410ef-c33f-4023-80db-3810af5ee365 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9cea756f95e843b5868e53f6446d648a', 'request_id': 'req-2b1410ef-c33f-4023-80db-3810af5ee365'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:14:38.226343 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-2b1410ef-c33f-4023-80db-3810af5ee365 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9cea756f95e843b5868e53f6446d648a', 'request_id': 'req-2b1410ef-c33f-4023-80db-3810af5ee365'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:14:38.234995 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-2b1410ef-c33f-4023-80db-3810af5ee365 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9cea756f95e843b5868e53f6446d648a', 'request_id': 'req-2b1410ef-c33f-4023-80db-3810af5ee365'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:14:38.263091 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-2b1410ef-c33f-4023-80db-3810af5ee365 admin admin] The token cache is being invalidate because project 9cea756f95e843b5868e53f6446d648a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81625) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:14:38.263303 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-2b1410ef-c33f-4023-80db-3810af5ee365 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:14:38.264061 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-2b1410ef-c33f-4023-80db-3810af5ee365 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:14:38.267933 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1226/2418] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:14:37 2022] DELETE /identity/v3/projects/9cea756f95e843b5868e53f6446d648a => generated 0 bytes in 292 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:14:38.274076 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54487bb8-0870-4fc2-ae0c-5f70b402aea4 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:14:38.274450 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54487bb8-0870-4fc2-ae0c-5f70b402aea4 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:14:38.274803 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54487bb8-0870-4fc2-ae0c-5f70b402aea4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:14:38.300465 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-54487bb8-0870-4fc2-ae0c-5f70b402aea4 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:14:38.302027 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-54487bb8-0870-4fc2-ae0c-5f70b402aea4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:14:38.370134 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:14:38.437257 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1193/2419] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:14:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:14:38.446806 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-144b1159-3205-443e-9b78-2d01815c2da6 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:14:38.447455 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-144b1159-3205-443e-9b78-2d01815c2da6 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:14:38.447757 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-144b1159-3205-443e-9b78-2d01815c2da6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:14:38.475648 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-144b1159-3205-443e-9b78-2d01815c2da6 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:14:38.476918 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-144b1159-3205-443e-9b78-2d01815c2da6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:14:38.488367 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1227/2420] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:14:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:14:38.492600 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54487bb8-0870-4fc2-ae0c-5f70b402aea4 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:14:38.508909 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54487bb8-0870-4fc2-ae0c-5f70b402aea4 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:14:38.513763 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5f14b4f-c4a0-4f22-a6d0-f72b7fe07d4f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:14:38.514556 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5f14b4f-c4a0-4f22-a6d0-f72b7fe07d4f admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:14:38.515021 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5f14b4f-c4a0-4f22-a6d0-f72b7fe07d4f admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:14:38.515443 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5f14b4f-c4a0-4f22-a6d0-f72b7fe07d4f admin admin] PATH_INFO: `/v3/domains` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:14:38.516262 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5f14b4f-c4a0-4f22-a6d0-f72b7fe07d4f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:14:38.516730 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5f14b4f-c4a0-4f22-a6d0-f72b7fe07d4f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:14:38.518483 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5f14b4f-c4a0-4f22-a6d0-f72b7fe07d4f admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:14:38.534080 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1194/2421] 10.222.0.77 () {58 vars in 1153 bytes} [Tue Aug 30 05:14:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:14:38.539240 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-144b1159-3205-443e-9b78-2d01815c2da6 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:14:38.553921 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-144b1159-3205-443e-9b78-2d01815c2da6 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:14:38.559895 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0b3cc69-59b0-43fe-b173-21cbbe21a978 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:14:38.560685 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0b3cc69-59b0-43fe-b173-21cbbe21a978 admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:14:38.569477 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0b3cc69-59b0-43fe-b173-21cbbe21a978 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:14:38.569992 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0b3cc69-59b0-43fe-b173-21cbbe21a978 admin admin] PATH_INFO: `/v3/projects` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:14:38.570857 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0b3cc69-59b0-43fe-b173-21cbbe21a978 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:14:38.573145 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0b3cc69-59b0-43fe-b173-21cbbe21a978 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:14:38.631970 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1228/2422] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:14:38 2022] POST /identity/v3/projects => generated 379 bytes in 89 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:14:38.633308 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5f14b4f-c4a0-4f22-a6d0-f72b7fe07d4f admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:14:38.650847 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5f14b4f-c4a0-4f22-a6d0-f72b7fe07d4f admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:14:38.736158 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd3ade25-5abd-474a-9241-b9de9297016c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:14:38.736802 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd3ade25-5abd-474a-9241-b9de9297016c admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:14:38.737085 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd3ade25-5abd-474a-9241-b9de9297016c admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:14:38.737380 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd3ade25-5abd-474a-9241-b9de9297016c admin admin] PATH_INFO: `/v3/users` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:14:38.738178 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd3ade25-5abd-474a-9241-b9de9297016c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:14:38.741044 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd3ade25-5abd-474a-9241-b9de9297016c admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:14:38.774160 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1195/2423] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:14:38 2022] POST /identity/v3/users => generated 290 bytes in 141 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 05:14:38.776270 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0b3cc69-59b0-43fe-b173-21cbbe21a978 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:14:38.793244 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0b3cc69-59b0-43fe-b173-21cbbe21a978 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:14:38.803115 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b4e086c-737f-4501-abe1-748831dd4eff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:14:38.803948 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b4e086c-737f-4501-abe1-748831dd4eff admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:14:38.804793 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b4e086c-737f-4501-abe1-748831dd4eff admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:14:38.805167 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b4e086c-737f-4501-abe1-748831dd4eff admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:14:38.805922 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b4e086c-737f-4501-abe1-748831dd4eff admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:14:38.808158 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b4e086c-737f-4501-abe1-748831dd4eff admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:14:38.822079 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1229/2424] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:14:38 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:14:38.824112 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd3ade25-5abd-474a-9241-b9de9297016c admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:14:38.836309 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd3ade25-5abd-474a-9241-b9de9297016c admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:14:38.844939 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-625e1df2-d4d3-47a9-8578-7f6371e5dbba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:14:38.845797 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-625e1df2-d4d3-47a9-8578-7f6371e5dbba admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:14:38.846224 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-625e1df2-d4d3-47a9-8578-7f6371e5dbba admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:14:38.846769 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-625e1df2-d4d3-47a9-8578-7f6371e5dbba admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:14:38.847501 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-625e1df2-d4d3-47a9-8578-7f6371e5dbba admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:14:38.849274 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-625e1df2-d4d3-47a9-8578-7f6371e5dbba admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:14:38.863854 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1196/2425] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:14:38 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:14:38.867928 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b4e086c-737f-4501-abe1-748831dd4eff admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:14:38.879285 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b4e086c-737f-4501-abe1-748831dd4eff admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:14:38.885805 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a42cc967-1943-4720-97cf-6dcb20b08542 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:14:38.886546 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a42cc967-1943-4720-97cf-6dcb20b08542 admin admin] REQUEST_METHOD: `PUT` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:14:38.886960 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a42cc967-1943-4720-97cf-6dcb20b08542 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:14:38.887346 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a42cc967-1943-4720-97cf-6dcb20b08542 admin admin] PATH_INFO: `/v3/projects/97e3452b02294219ae56cd23eb2aa0ce/users/9256fd43e5124fb68d4e5453584d1e8d/roles/6f03d86980c34c36b44a8d6764ff4fda` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:14:38.930896 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a42cc967-1943-4720-97cf-6dcb20b08542 admin admin] RBAC: Authorizing `identity:create_grant(project_id=97e3452b02294219ae56cd23eb2aa0ce, user_id=9256fd43e5124fb68d4e5453584d1e8d, role_id=6f03d86980c34c36b44a8d6764ff4fda)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:14:38.935767 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a42cc967-1943-4720-97cf-6dcb20b08542 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:14:38.962456 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1230/2426] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:14:38 2022] PUT /identity/v3/projects/97e3452b02294219ae56cd23eb2aa0ce/users/9256fd43e5124fb68d4e5453584d1e8d/roles/6f03d86980c34c36b44a8d6764ff4fda => generated 0 bytes in 95 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:14:38.967625 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb6cac35-be4c-4948-9901-40e75d27a3d5 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:14:38.967897 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb6cac35-be4c-4948-9901-40e75d27a3d5 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:14:38.968463 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb6cac35-be4c-4948-9901-40e75d27a3d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:14:38.993927 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-bb6cac35-be4c-4948-9901-40e75d27a3d5 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:14:38.995717 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-bb6cac35-be4c-4948-9901-40e75d27a3d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:14:39.150526 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1197/2427] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:14:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:14:39.166411 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a42cc967-1943-4720-97cf-6dcb20b08542 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:14:39.169020 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-a42cc967-1943-4720-97cf-6dcb20b08542 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:14:39.269712 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a42cc967-1943-4720-97cf-6dcb20b08542 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:14:39.277435 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e0cafb4-b9d2-40c8-8b5b-d60cc1f456eb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:14:39.278097 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e0cafb4-b9d2-40c8-8b5b-d60cc1f456eb service neutron] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:14:39.278383 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e0cafb4-b9d2-40c8-8b5b-d60cc1f456eb service neutron] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:14:39.278647 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e0cafb4-b9d2-40c8-8b5b-d60cc1f456eb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:14:39.290432 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e0cafb4-b9d2-40c8-8b5b-d60cc1f456eb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:14:39.291538 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e0cafb4-b9d2-40c8-8b5b-d60cc1f456eb service neutron] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:14:39.318873 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1231/2428] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:14:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 153 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:14:39.733984 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1b153a5-438b-4c58-bb68-f86d1f2385d8 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:14:39.733984 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1b153a5-438b-4c58-bb68-f86d1f2385d8 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:14:39.734723 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1b153a5-438b-4c58-bb68-f86d1f2385d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:14:39.760521 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-a1b153a5-438b-4c58-bb68-f86d1f2385d8 None None] MFA Rules not processed for user `31cc6aa564be4c569e6fa89813cb9329`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:14:39.762917 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-a1b153a5-438b-4c58-bb68-f86d1f2385d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:14:39.810096 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:14:39.836990 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1198/2429] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:14:39 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:14:45.853004 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a62a350-8b7b-4775-a586-7570e73e4231 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:14:45.854109 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a62a350-8b7b-4775-a586-7570e73e4231 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:14:45.854544 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a62a350-8b7b-4775-a586-7570e73e4231 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:14:45.895567 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-7a62a350-8b7b-4775-a586-7570e73e4231 None None] MFA Rules not processed for user `9256fd43e5124fb68d4e5453584d1e8d`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:14:45.897837 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-7a62a350-8b7b-4775-a586-7570e73e4231 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:14:46.062895 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=93,keystone:DELETE=4,keystone:INSERT=3 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:14:46.075912 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1232/2430] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:14:45 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 220 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:14:46.077149 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1b153a5-438b-4c58-bb68-f86d1f2385d8 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:14:46.095705 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1b153a5-438b-4c58-bb68-f86d1f2385d8 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:14:46.103660 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d275299-bde1-4126-92ce-bd9a7bc44c24 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:14:46.104284 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d275299-bde1-4126-92ce-bd9a7bc44c24 admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:14:46.104561 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d275299-bde1-4126-92ce-bd9a7bc44c24 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:14:46.106649 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d275299-bde1-4126-92ce-bd9a7bc44c24 admin admin] PATH_INFO: `/v3/projects` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:14:46.107356 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d275299-bde1-4126-92ce-bd9a7bc44c24 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:14:46.110996 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d275299-bde1-4126-92ce-bd9a7bc44c24 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:14:46.215011 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1199/2431] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:14:46 2022] POST /identity/v3/projects => generated 379 bytes in 138 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:14:46.215693 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a62a350-8b7b-4775-a586-7570e73e4231 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:14:46.243549 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a62a350-8b7b-4775-a586-7570e73e4231 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:14:46.359579 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0713fc1f-7ee6-43a9-8cdd-944ac4728dc3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:14:46.360213 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0713fc1f-7ee6-43a9-8cdd-944ac4728dc3 admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:14:46.360492 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0713fc1f-7ee6-43a9-8cdd-944ac4728dc3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:14:46.360932 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0713fc1f-7ee6-43a9-8cdd-944ac4728dc3 admin admin] PATH_INFO: `/v3/users` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:14:46.361398 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0713fc1f-7ee6-43a9-8cdd-944ac4728dc3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:14:46.364618 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0713fc1f-7ee6-43a9-8cdd-944ac4728dc3 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:14:46.398872 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1233/2432] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:14:46 2022] POST /identity/v3/users => generated 296 bytes in 184 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:14:46.403518 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d275299-bde1-4126-92ce-bd9a7bc44c24 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:14:46.415910 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d275299-bde1-4126-92ce-bd9a7bc44c24 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:14:46.427212 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80532aa1-aa1c-488d-9378-600804b93a25 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:14:46.428108 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80532aa1-aa1c-488d-9378-600804b93a25 admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:14:46.429048 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80532aa1-aa1c-488d-9378-600804b93a25 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:14:46.429495 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80532aa1-aa1c-488d-9378-600804b93a25 admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:14:46.430786 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80532aa1-aa1c-488d-9378-600804b93a25 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:14:46.433211 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80532aa1-aa1c-488d-9378-600804b93a25 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:14:46.447392 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1200/2433] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:14:46 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:14:46.451586 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0713fc1f-7ee6-43a9-8cdd-944ac4728dc3 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:14:46.462586 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0713fc1f-7ee6-43a9-8cdd-944ac4728dc3 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:14:46.490422 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fddcf21-8329-4fb1-be49-9a0e72b546f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:14:46.491444 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fddcf21-8329-4fb1-be49-9a0e72b546f5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:14:46.491875 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fddcf21-8329-4fb1-be49-9a0e72b546f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:14:46.492343 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fddcf21-8329-4fb1-be49-9a0e72b546f5 admin admin] PATH_INFO: `/v3/projects/3e3469ca8e2641179d0fc7988ed0e5c9/users/910c0e6f6a8646cf8671f1efc96424e1/roles/9d9d5d2c270146488249b7ed943280db` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:14:46.523571 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fddcf21-8329-4fb1-be49-9a0e72b546f5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3e3469ca8e2641179d0fc7988ed0e5c9, user_id=910c0e6f6a8646cf8671f1efc96424e1, role_id=9d9d5d2c270146488249b7ed943280db)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:14:46.527233 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fddcf21-8329-4fb1-be49-9a0e72b546f5 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:14:46.552970 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1234/2434] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:14:46 2022] PUT /identity/v3/projects/3e3469ca8e2641179d0fc7988ed0e5c9/users/910c0e6f6a8646cf8671f1efc96424e1/roles/9d9d5d2c270146488249b7ed943280db => generated 0 bytes in 99 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:14:46.554440 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80532aa1-aa1c-488d-9378-600804b93a25 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:14:46.565590 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80532aa1-aa1c-488d-9378-600804b93a25 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:14:46.657395 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aa265c8-c17b-4e9e-8de2-0f5e7a4058eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:14:46.658547 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aa265c8-c17b-4e9e-8de2-0f5e7a4058eb admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:14:46.659064 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aa265c8-c17b-4e9e-8de2-0f5e7a4058eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:14:46.659406 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aa265c8-c17b-4e9e-8de2-0f5e7a4058eb admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:14:46.660137 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aa265c8-c17b-4e9e-8de2-0f5e7a4058eb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:14:46.662399 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aa265c8-c17b-4e9e-8de2-0f5e7a4058eb admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:14:46.693873 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1201/2435] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:14:46 2022] GET /identity/v3/roles => generated 1420 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:14:46.697897 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fddcf21-8329-4fb1-be49-9a0e72b546f5 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:14:46.716532 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fddcf21-8329-4fb1-be49-9a0e72b546f5 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:14:46.724937 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bbbcfbb-a677-495e-876c-63adc539e5b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:14:46.725614 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bbbcfbb-a677-495e-876c-63adc539e5b8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:14:46.725893 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bbbcfbb-a677-495e-876c-63adc539e5b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:14:46.726182 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bbbcfbb-a677-495e-876c-63adc539e5b8 admin admin] PATH_INFO: `/v3/projects/3e3469ca8e2641179d0fc7988ed0e5c9/users/910c0e6f6a8646cf8671f1efc96424e1/roles/9d9d5d2c270146488249b7ed943280db` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:14:46.731816 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bbbcfbb-a677-495e-876c-63adc539e5b8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3e3469ca8e2641179d0fc7988ed0e5c9, user_id=910c0e6f6a8646cf8671f1efc96424e1, role_id=9d9d5d2c270146488249b7ed943280db)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:14:46.737578 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bbbcfbb-a677-495e-876c-63adc539e5b8 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:14:46.750784 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1235/2436] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:14:46 2022] PUT /identity/v3/projects/3e3469ca8e2641179d0fc7988ed0e5c9/users/910c0e6f6a8646cf8671f1efc96424e1/roles/9d9d5d2c270146488249b7ed943280db => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:14:49.312772 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aa265c8-c17b-4e9e-8de2-0f5e7a4058eb admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:14:49.315269 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-7aa265c8-c17b-4e9e-8de2-0f5e7a4058eb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:14:49.341139 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aa265c8-c17b-4e9e-8de2-0f5e7a4058eb admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:14:49.614629 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41e1ea99-0dd1-4d6a-8751-b61a76073429 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:14:49.615553 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e1ea99-0dd1-4d6a-8751-b61a76073429 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:14:49.619114 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e1ea99-0dd1-4d6a-8751-b61a76073429 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:14:49.619369 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e1ea99-0dd1-4d6a-8751-b61a76073429 admin admin] PATH_INFO: `/v3/users/2627cbebc4c040f4adba8d6f97cace78` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:14:49.620982 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41e1ea99-0dd1-4d6a-8751-b61a76073429 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2627cbebc4c040f4adba8d6f97cace78)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:14:49.626806 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41e1ea99-0dd1-4d6a-8751-b61a76073429 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:14:49.800739 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-41e1ea99-0dd1-4d6a-8751-b61a76073429 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2627cbebc4c040f4adba8d6f97cace78'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:14:49.801420 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-41e1ea99-0dd1-4d6a-8751-b61a76073429 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2627cbebc4c040f4adba8d6f97cace78'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:14:49.802051 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-41e1ea99-0dd1-4d6a-8751-b61a76073429 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2627cbebc4c040f4adba8d6f97cace78'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:14:49.810170 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=92,keystone:INSERT=1,keystone:DELETE=7 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:14:49.826017 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-41e1ea99-0dd1-4d6a-8751-b61a76073429 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2627cbebc4c040f4adba8d6f97cace78'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:14:49.835852 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-41e1ea99-0dd1-4d6a-8751-b61a76073429 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2627cbebc4c040f4adba8d6f97cace78'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:14:49.867595 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1202/2437] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:14:49 2022] DELETE /identity/v3/users/2627cbebc4c040f4adba8d6f97cace78 => generated 0 bytes in 554 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:14:49.871273 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0c0f9bc-096b-4bb9-8912-08d1681238f7 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:14:49.871748 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0c0f9bc-096b-4bb9-8912-08d1681238f7 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:14:49.871852 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0c0f9bc-096b-4bb9-8912-08d1681238f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:14:49.917019 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-e0c0f9bc-096b-4bb9-8912-08d1681238f7 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:14:49.925720 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-e0c0f9bc-096b-4bb9-8912-08d1681238f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:14:50.074001 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1236/2438] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:14:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 204 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:14:50.095375 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41e1ea99-0dd1-4d6a-8751-b61a76073429 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:14:50.098418 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-41e1ea99-0dd1-4d6a-8751-b61a76073429 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:14:50.218828 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41e1ea99-0dd1-4d6a-8751-b61a76073429 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:14:50.227315 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9b1ea3f-c97c-4cbe-b970-90888caba374 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:14:50.227894 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9b1ea3f-c97c-4cbe-b970-90888caba374 service neutron] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:14:50.229364 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9b1ea3f-c97c-4cbe-b970-90888caba374 service neutron] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:14:50.229647 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9b1ea3f-c97c-4cbe-b970-90888caba374 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:14:50.243378 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9b1ea3f-c97c-4cbe-b970-90888caba374 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:14:50.245193 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9b1ea3f-c97c-4cbe-b970-90888caba374 service neutron] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:14:50.262417 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1203/2439] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:14:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:14:50.822923 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0c0f9bc-096b-4bb9-8912-08d1681238f7 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:14:50.824772 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-e0c0f9bc-096b-4bb9-8912-08d1681238f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:14:50.840471 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0c0f9bc-096b-4bb9-8912-08d1681238f7 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:14:50.848818 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbcdcf47-6456-4941-883f-22f0aa0d74ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:14:50.849657 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbcdcf47-6456-4941-883f-22f0aa0d74ef admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:14:50.850756 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbcdcf47-6456-4941-883f-22f0aa0d74ef admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:14:50.851131 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbcdcf47-6456-4941-883f-22f0aa0d74ef admin admin] PATH_INFO: `/v3/projects/880e949296cb40018a76ad37b278e324` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:14:50.852714 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbcdcf47-6456-4941-883f-22f0aa0d74ef admin admin] RBAC: Authorizing `identity:delete_project(project_id=880e949296cb40018a76ad37b278e324)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:14:50.855055 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbcdcf47-6456-4941-883f-22f0aa0d74ef admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:14:51.018383 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=92,keystone:INSERT=5,keystone:DELETE=3 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:14:51.028966 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-cbcdcf47-6456-4941-883f-22f0aa0d74ef admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '880e949296cb40018a76ad37b278e324', 'request_id': 'req-cbcdcf47-6456-4941-883f-22f0aa0d74ef'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:14:51.059302 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-cbcdcf47-6456-4941-883f-22f0aa0d74ef admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '880e949296cb40018a76ad37b278e324', 'request_id': 'req-cbcdcf47-6456-4941-883f-22f0aa0d74ef'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:14:51.072366 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-cbcdcf47-6456-4941-883f-22f0aa0d74ef admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '880e949296cb40018a76ad37b278e324', 'request_id': 'req-cbcdcf47-6456-4941-883f-22f0aa0d74ef'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:14:51.100211 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-cbcdcf47-6456-4941-883f-22f0aa0d74ef admin admin] The token cache is being invalidate because project 880e949296cb40018a76ad37b278e324 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81625) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:14:51.101793 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-cbcdcf47-6456-4941-883f-22f0aa0d74ef admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:14:51.103873 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-cbcdcf47-6456-4941-883f-22f0aa0d74ef admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:14:51.106328 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1237/2440] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:14:50 2022] DELETE /identity/v3/projects/880e949296cb40018a76ad37b278e324 => generated 0 bytes in 284 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:14:53.985554 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d8eac84-ee66-4dd1-8472-3d58d89c2e62 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:14:53.986047 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d8eac84-ee66-4dd1-8472-3d58d89c2e62 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:14:53.986047 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d8eac84-ee66-4dd1-8472-3d58d89c2e62 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:14:54.019211 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-8d8eac84-ee66-4dd1-8472-3d58d89c2e62 None None] MFA Rules not processed for user `910c0e6f6a8646cf8671f1efc96424e1`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:14:54.020876 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-8d8eac84-ee66-4dd1-8472-3d58d89c2e62 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:14:54.116386 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1204/2441] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:14:53 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:14:54.130591 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbcdcf47-6456-4941-883f-22f0aa0d74ef admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:14:54.132161 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-cbcdcf47-6456-4941-883f-22f0aa0d74ef admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:14:54.221360 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbcdcf47-6456-4941-883f-22f0aa0d74ef admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:14:54.228253 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78c870f2-a18a-4bc5-8673-77fa64781769 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '886eb5e00bbf4ec292eb962b665b37d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:14:54.229082 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78c870f2-a18a-4bc5-8673-77fa64781769 service nova] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:14:54.229649 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78c870f2-a18a-4bc5-8673-77fa64781769 service nova] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:14:54.230040 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78c870f2-a18a-4bc5-8673-77fa64781769 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:14:54.232455 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-78c870f2-a18a-4bc5-8673-77fa64781769 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:14:54.301140 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78c870f2-a18a-4bc5-8673-77fa64781769 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:14:54.302793 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78c870f2-a18a-4bc5-8673-77fa64781769 service nova] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:14:54.314146 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1238/2442] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:14:54 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 184 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:15:00.202618 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d8eac84-ee66-4dd1-8472-3d58d89c2e62 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:15:00.204081 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-8d8eac84-ee66-4dd1-8472-3d58d89c2e62 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:15:00.265199 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:DELETE=3,keystone:SELECT=96,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:15:00.281614 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d8eac84-ee66-4dd1-8472-3d58d89c2e62 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:15:00.288208 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23515100-e203-4b75-b638-20a0f1b3b534 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1972959402704b89925001cd8d90cb0e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:15:00.288813 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23515100-e203-4b75-b638-20a0f1b3b534 service placement] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:15:00.289088 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23515100-e203-4b75-b638-20a0f1b3b534 service placement] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:15:00.289354 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23515100-e203-4b75-b638-20a0f1b3b534 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:15:00.291053 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-23515100-e203-4b75-b638-20a0f1b3b534 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:15:00.303221 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23515100-e203-4b75-b638-20a0f1b3b534 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:15:00.304853 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23515100-e203-4b75-b638-20a0f1b3b534 service placement] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:15:00.314885 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1205/2443] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:15:00 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 113 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:15:01.771905 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-480b6bcb-c04a-4b48-8788-674104db92bf None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:15:01.772383 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-480b6bcb-c04a-4b48-8788-674104db92bf None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:15:01.772462 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-480b6bcb-c04a-4b48-8788-674104db92bf None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:15:01.773748 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1239/2444] 10.222.0.77 () {54 vars in 921 bytes} [Tue Aug 30 05:15:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:15:01.780212 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e24291fa-1f2b-49d1-b1db-12bf1e1d1820 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:15:01.780653 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e24291fa-1f2b-49d1-b1db-12bf1e1d1820 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:15:01.780764 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e24291fa-1f2b-49d1-b1db-12bf1e1d1820 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:15:01.802590 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-e24291fa-1f2b-49d1-b1db-12bf1e1d1820 None None] MFA Rules not processed for user `886eb5e00bbf4ec292eb962b665b37d0`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:15:01.803434 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-e24291fa-1f2b-49d1-b1db-12bf1e1d1820 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:15:01.812792 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1206/2445] 10.222.0.77 () {58 vars in 1018 bytes} [Tue Aug 30 05:15:01 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:15:01.824817 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-480b6bcb-c04a-4b48-8788-674104db92bf None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:15:01.826644 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-480b6bcb-c04a-4b48-8788-674104db92bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:15:01.892283 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-480b6bcb-c04a-4b48-8788-674104db92bf None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:15:01.896243 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-494ff8d3-bf73-4cb2-afce-121633e7aedd service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a002ac0bf240a891ac7304178d3f0e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:15:01.897032 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-494ff8d3-bf73-4cb2-afce-121633e7aedd service glance] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:15:01.897320 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-494ff8d3-bf73-4cb2-afce-121633e7aedd service glance] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:15:01.897568 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-494ff8d3-bf73-4cb2-afce-121633e7aedd service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:15:01.907934 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-494ff8d3-bf73-4cb2-afce-121633e7aedd service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:15:01.909363 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-494ff8d3-bf73-4cb2-afce-121633e7aedd service glance] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:15:01.925240 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1240/2446] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:15:01 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 05:15:01.999808 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9c145a7-8a39-4398-81d3-27cccc15ca04 None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:15:02.000090 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9c145a7-8a39-4398-81d3-27cccc15ca04 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:15:02.000359 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9c145a7-8a39-4398-81d3-27cccc15ca04 None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:15:02.001690 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1207/2447] 10.222.0.77 () {54 vars in 921 bytes} [Tue Aug 30 05:15:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:15:02.006810 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca5dce47-7137-4573-90a3-a2823259c4cf None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:15:02.007143 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca5dce47-7137-4573-90a3-a2823259c4cf None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:15:02.007419 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca5dce47-7137-4573-90a3-a2823259c4cf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:15:02.028621 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-ca5dce47-7137-4573-90a3-a2823259c4cf None None] MFA Rules not processed for user `c33c45f4770d4151858e1d897d09af69`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:15:02.029872 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-ca5dce47-7137-4573-90a3-a2823259c4cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:15:02.039464 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:DELETE=4,keystone:SELECT=95,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:15:02.087630 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1241/2448] 10.222.0.77 () {58 vars in 1018 bytes} [Tue Aug 30 05:15:02 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:15:02.100317 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9c145a7-8a39-4398-81d3-27cccc15ca04 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:15:02.102031 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-b9c145a7-8a39-4398-81d3-27cccc15ca04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:15:02.117416 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9c145a7-8a39-4398-81d3-27cccc15ca04 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:15:02.121774 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71999c79-fc4b-4c0d-871f-b4e06863d6a6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:15:02.122418 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71999c79-fc4b-4c0d-871f-b4e06863d6a6 service neutron] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:15:02.122695 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71999c79-fc4b-4c0d-871f-b4e06863d6a6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:15:02.122992 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71999c79-fc4b-4c0d-871f-b4e06863d6a6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:15:02.133879 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71999c79-fc4b-4c0d-871f-b4e06863d6a6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:15:02.135575 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71999c79-fc4b-4c0d-871f-b4e06863d6a6 service neutron] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:15:02.145112 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1208/2449] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:15:02 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:15:02.230162 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-109bed9d-e28b-4fe6-a82d-b21641619d89 None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:15:02.230521 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-109bed9d-e28b-4fe6-a82d-b21641619d89 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:15:02.230910 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-109bed9d-e28b-4fe6-a82d-b21641619d89 None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:15:02.232490 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1242/2450] 10.222.0.77 () {54 vars in 921 bytes} [Tue Aug 30 05:15:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:15:02.239207 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b87a59-232d-4a8d-9e05-c4cd2a96b3c7 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:15:02.250916 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b87a59-232d-4a8d-9e05-c4cd2a96b3c7 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:15:02.250916 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b87a59-232d-4a8d-9e05-c4cd2a96b3c7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:15:02.262217 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-13b87a59-232d-4a8d-9e05-c4cd2a96b3c7 None None] MFA Rules not processed for user `886eb5e00bbf4ec292eb962b665b37d0`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:15:02.263443 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-13b87a59-232d-4a8d-9e05-c4cd2a96b3c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:15:02.275189 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1209/2451] 10.222.0.77 () {58 vars in 1018 bytes} [Tue Aug 30 05:15:02 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:15:02.287308 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-109bed9d-e28b-4fe6-a82d-b21641619d89 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:15:02.288904 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-109bed9d-e28b-4fe6-a82d-b21641619d89 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:15:02.303370 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-109bed9d-e28b-4fe6-a82d-b21641619d89 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:15:02.307341 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84e7b5b6-0405-4a4f-8dde-e3b053454e30 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:15:02.307945 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e7b5b6-0405-4a4f-8dde-e3b053454e30 service neutron] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:15:02.308254 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e7b5b6-0405-4a4f-8dde-e3b053454e30 service neutron] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:15:02.308525 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e7b5b6-0405-4a4f-8dde-e3b053454e30 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:15:02.318747 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84e7b5b6-0405-4a4f-8dde-e3b053454e30 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:15:02.320209 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84e7b5b6-0405-4a4f-8dde-e3b053454e30 service neutron] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:15:02.330909 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1243/2452] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:15:02 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:15:08.222432 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f3844b-ccd6-42fd-a061-588e04121290 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:15:08.222903 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f3844b-ccd6-42fd-a061-588e04121290 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:15:08.226241 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f3844b-ccd6-42fd-a061-588e04121290 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:15:08.256996 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-53f3844b-ccd6-42fd-a061-588e04121290 None None] MFA Rules not processed for user `fe51dbec91624ed1ab00b441e29a3dd5`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:15:08.259050 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-53f3844b-ccd6-42fd-a061-588e04121290 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:15:08.337194 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1210/2453] 10.222.0.77 () {58 vars in 1009 bytes} [Tue Aug 30 05:15:08 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:15:08.369570 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84e7b5b6-0405-4a4f-8dde-e3b053454e30 service neutron] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:15:08.372537 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-84e7b5b6-0405-4a4f-8dde-e3b053454e30 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:15:08.471100 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84e7b5b6-0405-4a4f-8dde-e3b053454e30 service neutron] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:15:08.478346 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe2130be-31a7-4afc-9880-c9cac8cf588e service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a7f7afc3d44ec788c05d005e52a1b8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:15:08.479130 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe2130be-31a7-4afc-9880-c9cac8cf588e service swift] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:15:08.479509 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe2130be-31a7-4afc-9880-c9cac8cf588e service swift] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:15:08.479917 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe2130be-31a7-4afc-9880-c9cac8cf588e service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:15:08.489413 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe2130be-31a7-4afc-9880-c9cac8cf588e service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:15:08.491198 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe2130be-31a7-4afc-9880-c9cac8cf588e service swift] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:15:08.500373 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1244/2454] 10.222.0.77 () {60 vars in 1384 bytes} [Tue Aug 30 05:15:08 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 131 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 30 05:15:18.257116 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5c50746-fdb4-4c9f-afa9-3c1f7bd9967f None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:15:18.257671 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5c50746-fdb4-4c9f-afa9-3c1f7bd9967f None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:15:18.257671 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5c50746-fdb4-4c9f-afa9-3c1f7bd9967f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:15:18.276392 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-b5c50746-fdb4-4c9f-afa9-3c1f7bd9967f None None] MFA Rules not processed for user `7522bcceac404e8aa10f719507fe1436`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:15:18.277592 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-b5c50746-fdb4-4c9f-afa9-3c1f7bd9967f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:15:18.335136 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1211/2455] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:15:18 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:15:18.496315 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=55 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:15:20.387603 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00884cb0-941f-4aaf-8d78-6054f06283d3 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:15:20.388223 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00884cb0-941f-4aaf-8d78-6054f06283d3 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:15:20.388223 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00884cb0-941f-4aaf-8d78-6054f06283d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:15:20.407432 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-00884cb0-941f-4aaf-8d78-6054f06283d3 None None] MFA Rules not processed for user `9256fd43e5124fb68d4e5453584d1e8d`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:15:20.408650 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-00884cb0-941f-4aaf-8d78-6054f06283d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:15:20.416294 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1245/2456] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:15:20 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:15:28.327973 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=68 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:15:30.402479 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=3 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:15:34.684730 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9412aba1-9ba3-4b77-9aee-735ef7084e9c None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:15:34.685234 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9412aba1-9ba3-4b77-9aee-735ef7084e9c None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:15:34.685353 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9412aba1-9ba3-4b77-9aee-735ef7084e9c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:15:34.705028 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-9412aba1-9ba3-4b77-9aee-735ef7084e9c None None] MFA Rules not processed for user `fe51dbec91624ed1ab00b441e29a3dd5`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:15:34.706213 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-9412aba1-9ba3-4b77-9aee-735ef7084e9c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:15:34.713131 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1212/2457] 10.222.0.77 () {58 vars in 967 bytes} [Tue Aug 30 05:15:34 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:15:34.725313 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00884cb0-941f-4aaf-8d78-6054f06283d3 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:15:34.737074 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00884cb0-941f-4aaf-8d78-6054f06283d3 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:15:34.742555 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afa02d12-ad06-40f7-a78e-3ff2e08fa9a2 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a7f7afc3d44ec788c05d005e52a1b8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:15:34.743188 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afa02d12-ad06-40f7-a78e-3ff2e08fa9a2 service swift] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:15:34.743467 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afa02d12-ad06-40f7-a78e-3ff2e08fa9a2 service swift] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:15:34.743732 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afa02d12-ad06-40f7-a78e-3ff2e08fa9a2 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:15:34.751675 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afa02d12-ad06-40f7-a78e-3ff2e08fa9a2 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:15:34.753264 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afa02d12-ad06-40f7-a78e-3ff2e08fa9a2 service swift] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:15:34.761880 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1246/2458] 10.222.0.77 () {60 vars in 1384 bytes} [Tue Aug 30 05:15:34 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 30 05:15:38.068609 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9412aba1-9ba3-4b77-9aee-735ef7084e9c None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:15:38.077890 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9412aba1-9ba3-4b77-9aee-735ef7084e9c None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:15:38.081393 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-828030ad-a108-4468-8a64-2caf7bd25ad5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:15:38.081976 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-828030ad-a108-4468-8a64-2caf7bd25ad5 service neutron] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:15:38.082255 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-828030ad-a108-4468-8a64-2caf7bd25ad5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:15:38.082527 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-828030ad-a108-4468-8a64-2caf7bd25ad5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:15:38.091227 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-828030ad-a108-4468-8a64-2caf7bd25ad5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:15:38.092898 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-828030ad-a108-4468-8a64-2caf7bd25ad5 service neutron] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:15:38.102614 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1213/2459] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:15:38 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 34 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:15:44.758787 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=6 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:15:48.098026 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=9 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:16:22.353251 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e0ad9d7-d146-474f-b100-0eac631dc970 None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:16:22.353805 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e0ad9d7-d146-474f-b100-0eac631dc970 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:16:22.353805 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e0ad9d7-d146-474f-b100-0eac631dc970 None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:16:22.355148 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1247/2460] 10.222.0.77 () {54 vars in 921 bytes} [Tue Aug 30 05:16:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:16:22.382797 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be43f680-de3e-4c81-bdab-83395d39c44e None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:16:22.383295 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be43f680-de3e-4c81-bdab-83395d39c44e None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:16:22.383450 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be43f680-de3e-4c81-bdab-83395d39c44e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:16:22.405678 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-be43f680-de3e-4c81-bdab-83395d39c44e None None] MFA Rules not processed for user `1972959402704b89925001cd8d90cb0e`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:16:22.407283 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-be43f680-de3e-4c81-bdab-83395d39c44e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:16:22.416230 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e0ad9d7-d146-474f-b100-0eac631dc970 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:16:22.418204 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-6e0ad9d7-d146-474f-b100-0eac631dc970 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:16:22.418897 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1214/2461] 10.222.0.77 () {58 vars in 1018 bytes} [Tue Aug 30 05:16:22 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 05:16:22.438566 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-378e53ec-2a7b-476a-a8ce-44c51f82e8d1 None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:16:22.438876 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-378e53ec-2a7b-476a-a8ce-44c51f82e8d1 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:16:22.439147 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-378e53ec-2a7b-476a-a8ce-44c51f82e8d1 None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:16:22.440696 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1215/2462] 10.222.0.77 () {54 vars in 961 bytes} [Tue Aug 30 05:16:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 05:16:22.445772 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a60028df-9d72-40e8-852a-836e4df73d4a None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:16:22.446217 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a60028df-9d72-40e8-852a-836e4df73d4a None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:16:22.446660 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a60028df-9d72-40e8-852a-836e4df73d4a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:16:22.471547 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-a60028df-9d72-40e8-852a-836e4df73d4a None None] MFA Rules not processed for user `1972959402704b89925001cd8d90cb0e`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:16:22.472948 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-a60028df-9d72-40e8-852a-836e4df73d4a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:16:22.481783 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1216/2463] 10.222.0.77 () {58 vars in 1058 bytes} [Tue Aug 30 05:16:22 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:16:22.491911 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a60028df-9d72-40e8-852a-836e4df73d4a None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:16:22.504067 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a60028df-9d72-40e8-852a-836e4df73d4a None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:16:22.509211 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da9cd038-9626-4647-8a31-6cdc0a07e6dd service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1972959402704b89925001cd8d90cb0e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:16:22.510083 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da9cd038-9626-4647-8a31-6cdc0a07e6dd service placement] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:16:22.510509 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da9cd038-9626-4647-8a31-6cdc0a07e6dd service placement] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:16:22.510647 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e0ad9d7-d146-474f-b100-0eac631dc970 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:16:22.511093 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da9cd038-9626-4647-8a31-6cdc0a07e6dd service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:16:22.515893 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12a56b03-7525-424c-86a8-deba57906083 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:16:22.516520 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12a56b03-7525-424c-86a8-deba57906083 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:16:22.516784 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12a56b03-7525-424c-86a8-deba57906083 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:16:22.517053 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12a56b03-7525-424c-86a8-deba57906083 admin admin] PATH_INFO: `/v3/users/7522bcceac404e8aa10f719507fe1436` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:16:22.518268 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12a56b03-7525-424c-86a8-deba57906083 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7522bcceac404e8aa10f719507fe1436)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:16:22.520252 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12a56b03-7525-424c-86a8-deba57906083 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:16:22.523769 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da9cd038-9626-4647-8a31-6cdc0a07e6dd service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:16:22.525656 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da9cd038-9626-4647-8a31-6cdc0a07e6dd service placement] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:16:22.538929 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1217/2464] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:16:22 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:16:22.572887 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-12a56b03-7525-424c-86a8-deba57906083 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7522bcceac404e8aa10f719507fe1436'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:16:22.581178 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-12a56b03-7525-424c-86a8-deba57906083 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7522bcceac404e8aa10f719507fe1436'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:16:22.594009 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-12a56b03-7525-424c-86a8-deba57906083 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7522bcceac404e8aa10f719507fe1436'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:16:22.600739 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-12a56b03-7525-424c-86a8-deba57906083 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7522bcceac404e8aa10f719507fe1436'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:16:22.601432 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-12a56b03-7525-424c-86a8-deba57906083 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7522bcceac404e8aa10f719507fe1436'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:16:22.607758 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1248/2465] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:16:22 2022] DELETE /identity/v3/users/7522bcceac404e8aa10f719507fe1436 => generated 0 bytes in 192 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 05:16:22.772468 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da9cd038-9626-4647-8a31-6cdc0a07e6dd service placement] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:16:22.785030 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da9cd038-9626-4647-8a31-6cdc0a07e6dd service placement] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:16:22.843020 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45eda132-9942-42fc-b1e9-7bcabb4959f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:16:22.843550 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45eda132-9942-42fc-b1e9-7bcabb4959f8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:16:22.843828 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45eda132-9942-42fc-b1e9-7bcabb4959f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:16:22.844096 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45eda132-9942-42fc-b1e9-7bcabb4959f8 admin admin] PATH_INFO: `/v3/projects/d98f56be5fb14ebe9e938ebc0f642994` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:16:22.845190 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45eda132-9942-42fc-b1e9-7bcabb4959f8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d98f56be5fb14ebe9e938ebc0f642994)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:16:22.847130 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45eda132-9942-42fc-b1e9-7bcabb4959f8 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:16:22.892933 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-45eda132-9942-42fc-b1e9-7bcabb4959f8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd98f56be5fb14ebe9e938ebc0f642994', 'request_id': 'req-45eda132-9942-42fc-b1e9-7bcabb4959f8'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:16:22.899436 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-45eda132-9942-42fc-b1e9-7bcabb4959f8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd98f56be5fb14ebe9e938ebc0f642994', 'request_id': 'req-45eda132-9942-42fc-b1e9-7bcabb4959f8'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:16:22.905682 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-45eda132-9942-42fc-b1e9-7bcabb4959f8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd98f56be5fb14ebe9e938ebc0f642994', 'request_id': 'req-45eda132-9942-42fc-b1e9-7bcabb4959f8'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:16:22.916946 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-45eda132-9942-42fc-b1e9-7bcabb4959f8 admin admin] The token cache is being invalidate because project d98f56be5fb14ebe9e938ebc0f642994 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81624) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:16:22.917237 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-45eda132-9942-42fc-b1e9-7bcabb4959f8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:16:22.917862 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-45eda132-9942-42fc-b1e9-7bcabb4959f8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:16:22.919854 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1218/2466] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:16:22 2022] DELETE /identity/v3/projects/d98f56be5fb14ebe9e938ebc0f642994 => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:16:32.559318 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-271cc02d-3491-49e3-8de5-5b7312299636 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:16:32.559928 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-271cc02d-3491-49e3-8de5-5b7312299636 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:16:32.559928 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-271cc02d-3491-49e3-8de5-5b7312299636 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:16:32.580533 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-271cc02d-3491-49e3-8de5-5b7312299636 None None] MFA Rules not processed for user `2fe451987a8c49c5b5fc0ef1c9c30ab5`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:16:32.581738 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-271cc02d-3491-49e3-8de5-5b7312299636 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:16:32.639024 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1249/2467] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:16:32 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:16:32.913319 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=63,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:16:38.210302 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8efd4607-81e8-4826-9867-fc2780c58c18 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:16:38.210991 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8efd4607-81e8-4826-9867-fc2780c58c18 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:16:38.210991 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8efd4607-81e8-4826-9867-fc2780c58c18 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:16:38.230485 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-8efd4607-81e8-4826-9867-fc2780c58c18 None None] MFA Rules not processed for user `31cc6aa564be4c569e6fa89813cb9329`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:16:38.231807 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-8efd4607-81e8-4826-9867-fc2780c58c18 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:16:38.286522 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1219/2468] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:16:38 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 77 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:16:42.631606 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=78,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:16:48.280035 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=25 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:16:48.562415 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5bc3c43-88d6-41ca-beff-09b5bbf6d225 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:16:48.562938 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5bc3c43-88d6-41ca-beff-09b5bbf6d225 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:16:48.563040 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5bc3c43-88d6-41ca-beff-09b5bbf6d225 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:16:48.581469 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-d5bc3c43-88d6-41ca-beff-09b5bbf6d225 None None] MFA Rules not processed for user `a8f01d0042c84767899dc2b71bca4e12`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:16:48.582663 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-d5bc3c43-88d6-41ca-beff-09b5bbf6d225 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:16:48.636776 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1250/2469] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:16:48 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 75 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:16:58.630237 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=25 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:17:20.230473 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8efd4607-81e8-4826-9867-fc2780c58c18 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:17:20.231907 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-8efd4607-81e8-4826-9867-fc2780c58c18 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:17:20.287456 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8efd4607-81e8-4826-9867-fc2780c58c18 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:17:20.290994 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-399f262d-b741-47bc-981f-33106ddcdb82 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '019a79b2affe478b8a30d1c538d0415c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:17:20.291581 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-399f262d-b741-47bc-981f-33106ddcdb82 service cinder] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:17:20.291858 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-399f262d-b741-47bc-981f-33106ddcdb82 service cinder] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:17:20.292125 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-399f262d-b741-47bc-981f-33106ddcdb82 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:17:20.293786 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-399f262d-b741-47bc-981f-33106ddcdb82 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:17:20.304928 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-399f262d-b741-47bc-981f-33106ddcdb82 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:17:20.306340 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-399f262d-b741-47bc-981f-33106ddcdb82 service cinder] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:17:20.315177 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1220/2470] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:17:20 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:17:20.426443 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5bc3c43-88d6-41ca-beff-09b5bbf6d225 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:17:20.427888 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-d5bc3c43-88d6-41ca-beff-09b5bbf6d225 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:17:20.439582 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5bc3c43-88d6-41ca-beff-09b5bbf6d225 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:17:20.443258 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-245d8a0a-193d-408a-92d2-2cb73f8a4ba3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '019a79b2affe478b8a30d1c538d0415c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:17:20.445117 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-245d8a0a-193d-408a-92d2-2cb73f8a4ba3 service cinder] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:17:20.445117 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-245d8a0a-193d-408a-92d2-2cb73f8a4ba3 service cinder] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:17:20.445117 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-245d8a0a-193d-408a-92d2-2cb73f8a4ba3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:17:20.446505 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-245d8a0a-193d-408a-92d2-2cb73f8a4ba3 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:17:20.516815 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-245d8a0a-193d-408a-92d2-2cb73f8a4ba3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:17:20.518197 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-245d8a0a-193d-408a-92d2-2cb73f8a4ba3 service cinder] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:17:20.530067 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1251/2471] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:17:20 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:17:23.159453 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-399f262d-b741-47bc-981f-33106ddcdb82 service cinder] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:17:23.160839 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-399f262d-b741-47bc-981f-33106ddcdb82 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:17:23.217328 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-399f262d-b741-47bc-981f-33106ddcdb82 service cinder] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:17:23.220819 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3fa36cf-db14-4c1e-829c-eb7541cc5599 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:17:23.221410 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3fa36cf-db14-4c1e-829c-eb7541cc5599 service neutron] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:17:23.221688 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3fa36cf-db14-4c1e-829c-eb7541cc5599 service neutron] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:17:23.221958 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3fa36cf-db14-4c1e-829c-eb7541cc5599 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:17:23.223669 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-f3fa36cf-db14-4c1e-829c-eb7541cc5599 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:17:23.237261 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3fa36cf-db14-4c1e-829c-eb7541cc5599 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:17:23.238692 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3fa36cf-db14-4c1e-829c-eb7541cc5599 service neutron] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:17:23.249865 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1221/2472] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:17:23 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:17:23.544651 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-245d8a0a-193d-408a-92d2-2cb73f8a4ba3 service cinder] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:17:23.546517 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-245d8a0a-193d-408a-92d2-2cb73f8a4ba3 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:17:23.560233 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-245d8a0a-193d-408a-92d2-2cb73f8a4ba3 service cinder] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:17:23.566282 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7decbea-e3cf-483c-9a73-35af293d3945 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '886eb5e00bbf4ec292eb962b665b37d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:17:23.566899 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7decbea-e3cf-483c-9a73-35af293d3945 service nova] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:17:23.567181 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7decbea-e3cf-483c-9a73-35af293d3945 service nova] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:17:23.567452 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7decbea-e3cf-483c-9a73-35af293d3945 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:17:23.569126 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-a7decbea-e3cf-483c-9a73-35af293d3945 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:17:23.579268 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7decbea-e3cf-483c-9a73-35af293d3945 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:17:23.580829 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7decbea-e3cf-483c-9a73-35af293d3945 service nova] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:17:23.589761 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1252/2473] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:17:23 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:17:33.244819 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=52 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:17:33.585586 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=40 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:17:34.130055 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3fa36cf-db14-4c1e-829c-eb7541cc5599 service neutron] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:17:34.132085 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-f3fa36cf-db14-4c1e-829c-eb7541cc5599 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:17:34.157381 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3fa36cf-db14-4c1e-829c-eb7541cc5599 service neutron] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:17:34.163606 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8866dc2a-b15e-4c1f-80a0-ffdb6d9cd099 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '886eb5e00bbf4ec292eb962b665b37d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:17:34.164271 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8866dc2a-b15e-4c1f-80a0-ffdb6d9cd099 service nova] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:17:34.164553 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8866dc2a-b15e-4c1f-80a0-ffdb6d9cd099 service nova] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:17:34.164824 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8866dc2a-b15e-4c1f-80a0-ffdb6d9cd099 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:17:34.166727 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-8866dc2a-b15e-4c1f-80a0-ffdb6d9cd099 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:17:34.180550 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8866dc2a-b15e-4c1f-80a0-ffdb6d9cd099 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:17:34.182155 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8866dc2a-b15e-4c1f-80a0-ffdb6d9cd099 service nova] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:17:34.193723 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1222/2474] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:17:34 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 64 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 05:17:37.195251 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7decbea-e3cf-483c-9a73-35af293d3945 service nova] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:17:37.206199 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7decbea-e3cf-483c-9a73-35af293d3945 service nova] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:17:37.211630 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e783f4a-3994-444b-a1b8-93c54f3854d8 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '886eb5e00bbf4ec292eb962b665b37d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:17:37.212208 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e783f4a-3994-444b-a1b8-93c54f3854d8 service nova] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:17:37.212507 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e783f4a-3994-444b-a1b8-93c54f3854d8 service nova] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:17:37.212775 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e783f4a-3994-444b-a1b8-93c54f3854d8 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:17:37.214484 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-7e783f4a-3994-444b-a1b8-93c54f3854d8 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:17:37.225088 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e783f4a-3994-444b-a1b8-93c54f3854d8 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:17:37.226636 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e783f4a-3994-444b-a1b8-93c54f3854d8 service nova] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:17:37.236258 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1253/2475] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:17:37 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:17:39.685751 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8866dc2a-b15e-4c1f-80a0-ffdb6d9cd099 service nova] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:17:39.687194 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-8866dc2a-b15e-4c1f-80a0-ffdb6d9cd099 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:17:39.698613 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8866dc2a-b15e-4c1f-80a0-ffdb6d9cd099 service nova] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:17:39.702890 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c3d344b-3647-4561-bbbd-401a76b08d50 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:17:39.703604 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c3d344b-3647-4561-bbbd-401a76b08d50 service neutron] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:17:39.703751 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c3d344b-3647-4561-bbbd-401a76b08d50 service neutron] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:17:39.704013 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c3d344b-3647-4561-bbbd-401a76b08d50 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:17:39.705667 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-0c3d344b-3647-4561-bbbd-401a76b08d50 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:17:39.770565 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c3d344b-3647-4561-bbbd-401a76b08d50 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:17:39.772083 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c3d344b-3647-4561-bbbd-401a76b08d50 service neutron] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:17:39.782762 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1223/2476] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:17:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:17:40.697066 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-720f5d77-dcec-4a4a-b54b-8d80e2692df3 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:17:40.698198 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-720f5d77-dcec-4a4a-b54b-8d80e2692df3 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:17:40.698594 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-720f5d77-dcec-4a4a-b54b-8d80e2692df3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:17:40.724118 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-720f5d77-dcec-4a4a-b54b-8d80e2692df3 None None] MFA Rules not processed for user `fe51dbec91624ed1ab00b441e29a3dd5`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:17:40.725728 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-720f5d77-dcec-4a4a-b54b-8d80e2692df3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:17:40.795982 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1254/2477] 10.222.0.77 () {58 vars in 1009 bytes} [Tue Aug 30 05:17:40 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:17:40.848679 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c3d344b-3647-4561-bbbd-401a76b08d50 service neutron] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:17:40.850408 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-0c3d344b-3647-4561-bbbd-401a76b08d50 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:17:40.918490 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c3d344b-3647-4561-bbbd-401a76b08d50 service neutron] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:17:40.923381 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbc2500f-838d-4386-82dd-ccc159eb49e1 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a7f7afc3d44ec788c05d005e52a1b8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:17:40.923979 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbc2500f-838d-4386-82dd-ccc159eb49e1 service swift] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:17:40.924252 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbc2500f-838d-4386-82dd-ccc159eb49e1 service swift] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:17:40.924533 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbc2500f-838d-4386-82dd-ccc159eb49e1 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:17:40.932581 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbc2500f-838d-4386-82dd-ccc159eb49e1 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:17:40.934186 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbc2500f-838d-4386-82dd-ccc159eb49e1 service swift] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:17:40.943065 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1224/2478] 10.222.0.77 () {60 vars in 1384 bytes} [Tue Aug 30 05:17:40 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 30 05:17:43.727869 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-720f5d77-dcec-4a4a-b54b-8d80e2692df3 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:17:43.729890 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-720f5d77-dcec-4a4a-b54b-8d80e2692df3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:17:43.755974 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-720f5d77-dcec-4a4a-b54b-8d80e2692df3 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:17:43.760895 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1433688-a664-414c-a1a1-690220dc07cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:17:43.761725 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1433688-a664-414c-a1a1-690220dc07cd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:17:43.762203 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1433688-a664-414c-a1a1-690220dc07cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:17:43.762477 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1433688-a664-414c-a1a1-690220dc07cd admin admin] PATH_INFO: `/v3/users/31cc6aa564be4c569e6fa89813cb9329` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:17:43.764205 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1433688-a664-414c-a1a1-690220dc07cd admin admin] RBAC: Authorizing `identity:delete_user(user_id=31cc6aa564be4c569e6fa89813cb9329)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:17:43.766135 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1433688-a664-414c-a1a1-690220dc07cd admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:17:43.820878 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-b1433688-a664-414c-a1a1-690220dc07cd admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '31cc6aa564be4c569e6fa89813cb9329'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:17:43.829332 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-b1433688-a664-414c-a1a1-690220dc07cd admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '31cc6aa564be4c569e6fa89813cb9329'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:17:43.845515 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-b1433688-a664-414c-a1a1-690220dc07cd admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '31cc6aa564be4c569e6fa89813cb9329'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:17:43.853042 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-b1433688-a664-414c-a1a1-690220dc07cd admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '31cc6aa564be4c569e6fa89813cb9329'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:17:43.853754 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-b1433688-a664-414c-a1a1-690220dc07cd admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '31cc6aa564be4c569e6fa89813cb9329'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:17:43.860537 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1255/2479] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:17:43 2022] DELETE /identity/v3/users/31cc6aa564be4c569e6fa89813cb9329 => generated 0 bytes in 133 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:17:44.068590 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbc2500f-838d-4386-82dd-ccc159eb49e1 service swift] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:17:44.090049 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbc2500f-838d-4386-82dd-ccc159eb49e1 service swift] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:17:44.159419 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c928560e-f6f4-463f-ba1d-73239eced766 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:17:44.160078 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c928560e-f6f4-463f-ba1d-73239eced766 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:17:44.160356 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c928560e-f6f4-463f-ba1d-73239eced766 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:17:44.160630 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c928560e-f6f4-463f-ba1d-73239eced766 admin admin] PATH_INFO: `/v3/projects/d2ce224c162c414c9be681b76801969c` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:17:44.161855 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c928560e-f6f4-463f-ba1d-73239eced766 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d2ce224c162c414c9be681b76801969c)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:17:44.163926 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c928560e-f6f4-463f-ba1d-73239eced766 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:17:44.177273 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:17:44.233958 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-c928560e-f6f4-463f-ba1d-73239eced766 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd2ce224c162c414c9be681b76801969c', 'request_id': 'req-c928560e-f6f4-463f-ba1d-73239eced766'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:17:44.262171 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-c928560e-f6f4-463f-ba1d-73239eced766 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd2ce224c162c414c9be681b76801969c', 'request_id': 'req-c928560e-f6f4-463f-ba1d-73239eced766'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:17:44.272195 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-c928560e-f6f4-463f-ba1d-73239eced766 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd2ce224c162c414c9be681b76801969c', 'request_id': 'req-c928560e-f6f4-463f-ba1d-73239eced766'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:17:44.283614 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-c928560e-f6f4-463f-ba1d-73239eced766 admin admin] The token cache is being invalidate because project d2ce224c162c414c9be681b76801969c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81624) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:17:44.283906 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-c928560e-f6f4-463f-ba1d-73239eced766 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:17:44.284573 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-c928560e-f6f4-463f-ba1d-73239eced766 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:17:44.286146 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1225/2480] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:17:44 2022] DELETE /identity/v3/projects/d2ce224c162c414c9be681b76801969c => generated 0 bytes in 218 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:17:44.291498 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-712161de-829b-4ed3-96b1-7680bf96a07e None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:17:44.292020 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-712161de-829b-4ed3-96b1-7680bf96a07e None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:17:44.292908 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-712161de-829b-4ed3-96b1-7680bf96a07e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:17:44.315897 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-712161de-829b-4ed3-96b1-7680bf96a07e None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:17:44.317644 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-712161de-829b-4ed3-96b1-7680bf96a07e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:17:44.393022 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1256/2481] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:17:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:17:44.401373 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e261d06-8d6d-4aa2-b03b-0e969da7f14f None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:17:44.401735 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e261d06-8d6d-4aa2-b03b-0e969da7f14f None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:17:44.403444 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e261d06-8d6d-4aa2-b03b-0e969da7f14f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:17:44.429108 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-9e261d06-8d6d-4aa2-b03b-0e969da7f14f None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:17:44.430428 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-9e261d06-8d6d-4aa2-b03b-0e969da7f14f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:17:44.448540 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1226/2482] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:17:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 48 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:17:44.449571 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-712161de-829b-4ed3-96b1-7680bf96a07e None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:17:44.460409 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:17:44.475899 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-712161de-829b-4ed3-96b1-7680bf96a07e None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:17:44.482238 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84f7327b-e548-42db-812d-93c1fd1d8c74 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:17:44.486938 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f7327b-e548-42db-812d-93c1fd1d8c74 admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:17:44.486938 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f7327b-e548-42db-812d-93c1fd1d8c74 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:17:44.486938 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f7327b-e548-42db-812d-93c1fd1d8c74 admin admin] PATH_INFO: `/v3/domains` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:17:44.486938 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84f7327b-e548-42db-812d-93c1fd1d8c74 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:17:44.486938 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84f7327b-e548-42db-812d-93c1fd1d8c74 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:17:44.486938 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84f7327b-e548-42db-812d-93c1fd1d8c74 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:17:44.507884 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1257/2483] 10.222.0.77 () {58 vars in 1153 bytes} [Tue Aug 30 05:17:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 59 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 05:17:44.508155 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e261d06-8d6d-4aa2-b03b-0e969da7f14f None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:17:44.518156 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e261d06-8d6d-4aa2-b03b-0e969da7f14f None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:17:44.523120 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f9fdbb7-31bf-43b7-b48d-3423038fcfba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:17:44.523935 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f9fdbb7-31bf-43b7-b48d-3423038fcfba admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:17:44.524132 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f9fdbb7-31bf-43b7-b48d-3423038fcfba admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:17:44.524477 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f9fdbb7-31bf-43b7-b48d-3423038fcfba admin admin] PATH_INFO: `/v3/projects` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:17:44.525403 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f9fdbb7-31bf-43b7-b48d-3423038fcfba admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:17:44.527434 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f9fdbb7-31bf-43b7-b48d-3423038fcfba admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:17:44.562606 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1227/2484] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:17:44 2022] POST /identity/v3/projects => generated 385 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 05:17:44.563805 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84f7327b-e548-42db-812d-93c1fd1d8c74 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:17:44.575950 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84f7327b-e548-42db-812d-93c1fd1d8c74 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:17:44.646783 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0a3d29e-e901-4bcf-b65f-7fe00b0adf95 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:17:44.647512 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0a3d29e-e901-4bcf-b65f-7fe00b0adf95 admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:17:44.647862 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0a3d29e-e901-4bcf-b65f-7fe00b0adf95 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:17:44.648227 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0a3d29e-e901-4bcf-b65f-7fe00b0adf95 admin admin] PATH_INFO: `/v3/users` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:17:44.648924 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0a3d29e-e901-4bcf-b65f-7fe00b0adf95 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:17:44.650979 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0a3d29e-e901-4bcf-b65f-7fe00b0adf95 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:17:44.679642 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1258/2485] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:17:44 2022] POST /identity/v3/users => generated 293 bytes in 116 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:17:44.683824 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f9fdbb7-31bf-43b7-b48d-3423038fcfba admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:17:44.696130 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f9fdbb7-31bf-43b7-b48d-3423038fcfba admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:17:44.701950 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-024ca0b7-854b-4664-a4ac-c3e04287c3c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:17:44.702632 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-024ca0b7-854b-4664-a4ac-c3e04287c3c2 admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:17:44.703029 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-024ca0b7-854b-4664-a4ac-c3e04287c3c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:17:44.703386 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-024ca0b7-854b-4664-a4ac-c3e04287c3c2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:17:44.704100 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-024ca0b7-854b-4664-a4ac-c3e04287c3c2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:17:44.705773 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-024ca0b7-854b-4664-a4ac-c3e04287c3c2 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:17:44.724986 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1228/2486] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:17:44 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:17:44.725905 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0a3d29e-e901-4bcf-b65f-7fe00b0adf95 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:17:44.740039 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0a3d29e-e901-4bcf-b65f-7fe00b0adf95 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:17:44.746938 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f1461b5-5dda-4a95-90d6-6f460c3493f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:17:44.747582 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f1461b5-5dda-4a95-90d6-6f460c3493f0 admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:17:44.747863 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f1461b5-5dda-4a95-90d6-6f460c3493f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:17:44.748133 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f1461b5-5dda-4a95-90d6-6f460c3493f0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:17:44.748978 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f1461b5-5dda-4a95-90d6-6f460c3493f0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:17:44.750748 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f1461b5-5dda-4a95-90d6-6f460c3493f0 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:17:44.765783 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1259/2487] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:17:44 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 05:17:44.769747 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-024ca0b7-854b-4664-a4ac-c3e04287c3c2 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:17:44.781351 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-024ca0b7-854b-4664-a4ac-c3e04287c3c2 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:17:44.787802 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27a48cac-db7a-49a7-a2af-02a31b5ee5c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:17:44.788838 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27a48cac-db7a-49a7-a2af-02a31b5ee5c1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:17:44.789524 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27a48cac-db7a-49a7-a2af-02a31b5ee5c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:17:44.789877 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27a48cac-db7a-49a7-a2af-02a31b5ee5c1 admin admin] PATH_INFO: `/v3/projects/bebd859c24cd4ed3b7e32a82de820b8a/users/d079e0264f0e40db9fc93b8f217a7a74/roles/6f03d86980c34c36b44a8d6764ff4fda` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:17:44.808143 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27a48cac-db7a-49a7-a2af-02a31b5ee5c1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bebd859c24cd4ed3b7e32a82de820b8a, user_id=d079e0264f0e40db9fc93b8f217a7a74, role_id=6f03d86980c34c36b44a8d6764ff4fda)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:17:44.810872 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27a48cac-db7a-49a7-a2af-02a31b5ee5c1 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:17:44.828658 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1229/2488] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:17:44 2022] PUT /identity/v3/projects/bebd859c24cd4ed3b7e32a82de820b8a/users/d079e0264f0e40db9fc93b8f217a7a74/roles/6f03d86980c34c36b44a8d6764ff4fda => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:17:44.833590 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f374a8-f02d-4ba4-893c-9e201ccdd3c2 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:17:44.833883 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f374a8-f02d-4ba4-893c-9e201ccdd3c2 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:17:44.834161 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f374a8-f02d-4ba4-893c-9e201ccdd3c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:17:44.857623 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-83f374a8-f02d-4ba4-893c-9e201ccdd3c2 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:17:44.859369 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-83f374a8-f02d-4ba4-893c-9e201ccdd3c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:17:44.927451 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1260/2489] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:17:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:17:44.937566 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27a48cac-db7a-49a7-a2af-02a31b5ee5c1 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:17:44.939462 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-27a48cac-db7a-49a7-a2af-02a31b5ee5c1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:17:45.019157 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27a48cac-db7a-49a7-a2af-02a31b5ee5c1 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:17:45.026457 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2748fb02-8f4c-4470-9d58-85afb8dda7b6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:17:45.027463 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2748fb02-8f4c-4470-9d58-85afb8dda7b6 service neutron] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:17:45.027887 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2748fb02-8f4c-4470-9d58-85afb8dda7b6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:17:45.028163 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2748fb02-8f4c-4470-9d58-85afb8dda7b6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:17:45.041751 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2748fb02-8f4c-4470-9d58-85afb8dda7b6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:17:45.045725 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2748fb02-8f4c-4470-9d58-85afb8dda7b6 service neutron] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:17:45.057070 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1230/2490] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:17:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 120 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:17:51.725776 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fbb4444-22f1-4250-85cc-eddf32037140 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:17:51.726288 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fbb4444-22f1-4250-85cc-eddf32037140 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:17:51.726388 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fbb4444-22f1-4250-85cc-eddf32037140 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:17:51.813384 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-7fbb4444-22f1-4250-85cc-eddf32037140 None None] MFA Rules not processed for user `d079e0264f0e40db9fc93b8f217a7a74`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:17:51.814644 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-7fbb4444-22f1-4250-85cc-eddf32037140 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:17:51.934000 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:17:51.943601 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1261/2491] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:17:51 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 219 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 05:17:51.962866 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2748fb02-8f4c-4470-9d58-85afb8dda7b6 service neutron] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:17:51.967038 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-2748fb02-8f4c-4470-9d58-85afb8dda7b6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:17:52.028379 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc9405a-fe94-4778-9f1c-d44bb651ab09 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:17:52.028709 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc9405a-fe94-4778-9f1c-d44bb651ab09 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:17:52.028996 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc9405a-fe94-4778-9f1c-d44bb651ab09 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:17:52.052464 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:17:52.059518 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-bbc9405a-fe94-4778-9f1c-d44bb651ab09 None None] MFA Rules not processed for user `9256fd43e5124fb68d4e5453584d1e8d`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:17:52.061326 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-bbc9405a-fe94-4778-9f1c-d44bb651ab09 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:17:52.111868 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2748fb02-8f4c-4470-9d58-85afb8dda7b6 service neutron] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:17:52.119744 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12b1313e-ac11-4b59-a6b0-166e5723770d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '886eb5e00bbf4ec292eb962b665b37d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:17:52.120564 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12b1313e-ac11-4b59-a6b0-166e5723770d service nova] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:17:52.120938 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12b1313e-ac11-4b59-a6b0-166e5723770d service nova] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:17:52.121345 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12b1313e-ac11-4b59-a6b0-166e5723770d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:17:52.135275 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12b1313e-ac11-4b59-a6b0-166e5723770d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:17:52.137106 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12b1313e-ac11-4b59-a6b0-166e5723770d service nova] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:17:52.158717 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1262/2492] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:17:52 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:17:52.170794 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1231/2493] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:17:51 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 209 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:17:52.813663 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbc9405a-fe94-4778-9f1c-d44bb651ab09 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:17:52.815252 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-bbc9405a-fe94-4778-9f1c-d44bb651ab09 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:17:52.831852 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbc9405a-fe94-4778-9f1c-d44bb651ab09 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:17:52.835639 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ade9c00-52ef-4ec6-bc69-d3c7ee4404ed service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:17:52.836261 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ade9c00-52ef-4ec6-bc69-d3c7ee4404ed service neutron] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:17:52.836539 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ade9c00-52ef-4ec6-bc69-d3c7ee4404ed service neutron] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:17:52.836830 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ade9c00-52ef-4ec6-bc69-d3c7ee4404ed service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:17:52.838569 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-7ade9c00-52ef-4ec6-bc69-d3c7ee4404ed service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:17:52.849360 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ade9c00-52ef-4ec6-bc69-d3c7ee4404ed service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:17:52.850857 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ade9c00-52ef-4ec6-bc69-d3c7ee4404ed service neutron] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:17:52.861407 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1263/2494] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:17:52 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 49 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:17:53.814070 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12b1313e-ac11-4b59-a6b0-166e5723770d service nova] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:17:53.815543 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-12b1313e-ac11-4b59-a6b0-166e5723770d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:17:54.020108 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12b1313e-ac11-4b59-a6b0-166e5723770d service nova] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:17:54.052581 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81b25448-58a5-4059-97ae-221a0e57d1d3 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1972959402704b89925001cd8d90cb0e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:17:54.053577 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81b25448-58a5-4059-97ae-221a0e57d1d3 service placement] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:17:54.054065 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81b25448-58a5-4059-97ae-221a0e57d1d3 service placement] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:17:54.054523 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81b25448-58a5-4059-97ae-221a0e57d1d3 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:17:54.056529 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-81b25448-58a5-4059-97ae-221a0e57d1d3 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:17:54.082163 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81b25448-58a5-4059-97ae-221a0e57d1d3 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:17:54.084481 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81b25448-58a5-4059-97ae-221a0e57d1d3 service placement] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:17:54.122705 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1232/2495] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:17:53 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:17:54.567945 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-828cd0e3-82ca-41e2-99c7-2212466b7c79 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:17:54.568860 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-828cd0e3-82ca-41e2-99c7-2212466b7c79 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:17:54.569291 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-828cd0e3-82ca-41e2-99c7-2212466b7c79 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:17:54.592785 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-828cd0e3-82ca-41e2-99c7-2212466b7c79 None None] MFA Rules not processed for user `d079e0264f0e40db9fc93b8f217a7a74`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:17:54.594479 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-828cd0e3-82ca-41e2-99c7-2212466b7c79 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:17:54.608471 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1264/2496] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:17:54 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:17:55.040919 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81b25448-58a5-4059-97ae-221a0e57d1d3 service placement] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:17:55.051177 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81b25448-58a5-4059-97ae-221a0e57d1d3 service placement] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:17:55.054705 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3128ab60-ec4b-4f2e-b61c-95578a9c7c40 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:17:55.055325 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3128ab60-ec4b-4f2e-b61c-95578a9c7c40 service neutron] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:17:55.055607 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3128ab60-ec4b-4f2e-b61c-95578a9c7c40 service neutron] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:17:55.055880 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3128ab60-ec4b-4f2e-b61c-95578a9c7c40 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:17:55.057583 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-3128ab60-ec4b-4f2e-b61c-95578a9c7c40 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:17:55.068641 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3128ab60-ec4b-4f2e-b61c-95578a9c7c40 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:17:55.070231 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3128ab60-ec4b-4f2e-b61c-95578a9c7c40 service neutron] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:17:55.079177 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1233/2497] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:17:55 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:17:56.297943 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-828cd0e3-82ca-41e2-99c7-2212466b7c79 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:17:56.313754 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-828cd0e3-82ca-41e2-99c7-2212466b7c79 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:17:56.317624 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2051c6f-ef4b-44db-8ed0-64750d972bd9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:17:56.318309 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2051c6f-ef4b-44db-8ed0-64750d972bd9 service neutron] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:17:56.318655 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2051c6f-ef4b-44db-8ed0-64750d972bd9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:17:56.318965 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2051c6f-ef4b-44db-8ed0-64750d972bd9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:17:56.320730 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-c2051c6f-ef4b-44db-8ed0-64750d972bd9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:17:56.332112 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2051c6f-ef4b-44db-8ed0-64750d972bd9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:17:56.333760 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2051c6f-ef4b-44db-8ed0-64750d972bd9 service neutron] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:17:56.343119 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1265/2498] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:17:56 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:17:59.660964 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3128ab60-ec4b-4f2e-b61c-95578a9c7c40 service neutron] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:17:59.662406 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-3128ab60-ec4b-4f2e-b61c-95578a9c7c40 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:17:59.678090 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3128ab60-ec4b-4f2e-b61c-95578a9c7c40 service neutron] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:17:59.683756 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d697ffde-1b11-4b20-8272-6b435df57075 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '886eb5e00bbf4ec292eb962b665b37d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:17:59.685031 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d697ffde-1b11-4b20-8272-6b435df57075 service nova] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:17:59.685200 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d697ffde-1b11-4b20-8272-6b435df57075 service nova] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:17:59.685468 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d697ffde-1b11-4b20-8272-6b435df57075 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:17:59.687241 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-d697ffde-1b11-4b20-8272-6b435df57075 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:17:59.704379 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d697ffde-1b11-4b20-8272-6b435df57075 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:17:59.706300 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d697ffde-1b11-4b20-8272-6b435df57075 service nova] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:17:59.718848 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1234/2499] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:17:59 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 58 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:18:01.669275 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad1eb20f-5447-412e-ace4-ede7fd59cf93 None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:01.669804 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad1eb20f-5447-412e-ace4-ede7fd59cf93 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:01.669804 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad1eb20f-5447-412e-ace4-ede7fd59cf93 None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:01.671185 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1266/2500] 10.222.0.77 () {54 vars in 921 bytes} [Tue Aug 30 05:18:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:18:01.677069 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56707ab2-09dc-43c3-b780-8c2410e9ffa6 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:01.677923 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56707ab2-09dc-43c3-b780-8c2410e9ffa6 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:01.678299 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56707ab2-09dc-43c3-b780-8c2410e9ffa6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:01.699180 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-56707ab2-09dc-43c3-b780-8c2410e9ffa6 None None] MFA Rules not processed for user `886eb5e00bbf4ec292eb962b665b37d0`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:18:01.700640 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-56707ab2-09dc-43c3-b780-8c2410e9ffa6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:18:01.710250 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1235/2501] 10.222.0.77 () {58 vars in 1018 bytes} [Tue Aug 30 05:18:01 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:18:01.723181 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad1eb20f-5447-412e-ace4-ede7fd59cf93 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:18:01.725133 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-ad1eb20f-5447-412e-ace4-ede7fd59cf93 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:18:01.788386 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad1eb20f-5447-412e-ace4-ede7fd59cf93 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:18:01.792499 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee1177da-2fc0-403f-ad9d-b7d45bfa9a2d service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a002ac0bf240a891ac7304178d3f0e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:18:01.793101 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee1177da-2fc0-403f-ad9d-b7d45bfa9a2d service glance] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:01.793375 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee1177da-2fc0-403f-ad9d-b7d45bfa9a2d service glance] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:01.793643 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee1177da-2fc0-403f-ad9d-b7d45bfa9a2d service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:01.804596 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee1177da-2fc0-403f-ad9d-b7d45bfa9a2d service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:18:01.806020 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee1177da-2fc0-403f-ad9d-b7d45bfa9a2d service glance] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:18:01.817875 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1267/2502] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:18:01 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:18:01.906134 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e22cde5-f45a-47c0-a3bd-9cf655f7d36c None None] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:01.907426 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e22cde5-f45a-47c0-a3bd-9cf655f7d36c None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:01.907709 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e22cde5-f45a-47c0-a3bd-9cf655f7d36c None None] PATH_INFO: `/` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:01.909089 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1236/2503] 10.222.0.77 () {54 vars in 921 bytes} [Tue Aug 30 05:18:01 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:18:01.914551 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c44909f-5e4d-4351-be42-3ccb824459cd None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:01.914871 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c44909f-5e4d-4351-be42-3ccb824459cd None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:01.915157 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c44909f-5e4d-4351-be42-3ccb824459cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:01.935137 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-5c44909f-5e4d-4351-be42-3ccb824459cd None None] MFA Rules not processed for user `c33c45f4770d4151858e1d897d09af69`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:18:01.936341 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-5c44909f-5e4d-4351-be42-3ccb824459cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:18:01.943000 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1268/2504] 10.222.0.77 () {58 vars in 1018 bytes} [Tue Aug 30 05:18:01 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:18:01.954652 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e22cde5-f45a-47c0-a3bd-9cf655f7d36c None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:18:01.966535 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e22cde5-f45a-47c0-a3bd-9cf655f7d36c None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:18:01.970035 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64554613-890b-4876-8074-1f97086efe55 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:18:01.970620 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64554613-890b-4876-8074-1f97086efe55 service neutron] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:01.970928 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64554613-890b-4876-8074-1f97086efe55 service neutron] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:01.971195 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64554613-890b-4876-8074-1f97086efe55 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:01.978182 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64554613-890b-4876-8074-1f97086efe55 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:18:01.979638 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64554613-890b-4876-8074-1f97086efe55 service neutron] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:18:01.988127 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1237/2505] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:18:01 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 34 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:18:02.092356 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df27e3d7-64f6-4fa3-8642-d378bd37e29d None None] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:02.092984 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df27e3d7-64f6-4fa3-8642-d378bd37e29d None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:02.093365 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df27e3d7-64f6-4fa3-8642-d378bd37e29d None None] PATH_INFO: `/` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:02.095153 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1269/2506] 10.222.0.77 () {54 vars in 921 bytes} [Tue Aug 30 05:18:02 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 05:18:02.102456 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa27feaf-6519-4616-a54a-95e0586aab5b None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:02.102742 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa27feaf-6519-4616-a54a-95e0586aab5b None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:02.103069 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa27feaf-6519-4616-a54a-95e0586aab5b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:02.135006 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-fa27feaf-6519-4616-a54a-95e0586aab5b None None] MFA Rules not processed for user `886eb5e00bbf4ec292eb962b665b37d0`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:18:02.136566 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-fa27feaf-6519-4616-a54a-95e0586aab5b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:18:02.147378 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1238/2507] 10.222.0.77 () {58 vars in 1018 bytes} [Tue Aug 30 05:18:02 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:18:02.164334 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df27e3d7-64f6-4fa3-8642-d378bd37e29d None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:18:02.176787 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df27e3d7-64f6-4fa3-8642-d378bd37e29d None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:18:02.180337 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8517f94-b499-4d04-8f37-7603a7086368 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:18:02.180935 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8517f94-b499-4d04-8f37-7603a7086368 service neutron] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:02.181209 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8517f94-b499-4d04-8f37-7603a7086368 service neutron] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:02.181478 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8517f94-b499-4d04-8f37-7603a7086368 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:02.191776 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8517f94-b499-4d04-8f37-7603a7086368 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:18:02.193212 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8517f94-b499-4d04-8f37-7603a7086368 service neutron] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:18:02.204008 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1270/2508] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:18:02 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:18:07.430588 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa27feaf-6519-4616-a54a-95e0586aab5b None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:18:07.443375 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa27feaf-6519-4616-a54a-95e0586aab5b None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:18:07.449596 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80b827e7-1cc6-4ed2-a130-457895580a6c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '886eb5e00bbf4ec292eb962b665b37d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:18:07.450770 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80b827e7-1cc6-4ed2-a130-457895580a6c service nova] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:07.451091 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80b827e7-1cc6-4ed2-a130-457895580a6c service nova] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:07.451391 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80b827e7-1cc6-4ed2-a130-457895580a6c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:07.453285 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-80b827e7-1cc6-4ed2-a130-457895580a6c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:18:07.473907 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80b827e7-1cc6-4ed2-a130-457895580a6c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:18:07.475575 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80b827e7-1cc6-4ed2-a130-457895580a6c service nova] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:18:07.486398 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1239/2509] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:18:07 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 56 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:18:07.604630 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23235fcd-aae2-4e40-8d05-da456936f93a None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:07.605126 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23235fcd-aae2-4e40-8d05-da456936f93a None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:07.605246 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23235fcd-aae2-4e40-8d05-da456936f93a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:07.629396 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-23235fcd-aae2-4e40-8d05-da456936f93a None None] MFA Rules not processed for user `fe51dbec91624ed1ab00b441e29a3dd5`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:18:07.630757 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-23235fcd-aae2-4e40-8d05-da456936f93a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:18:07.698093 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:18:07.709399 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1271/2510] 10.222.0.77 () {58 vars in 1009 bytes} [Tue Aug 30 05:18:07 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 05:18:07.730393 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80b827e7-1cc6-4ed2-a130-457895580a6c service nova] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:18:07.732727 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-80b827e7-1cc6-4ed2-a130-457895580a6c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:18:07.821778 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:18:07.842372 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80b827e7-1cc6-4ed2-a130-457895580a6c service nova] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:18:07.848071 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0df094a8-3416-4acd-9395-708713bd1948 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a7f7afc3d44ec788c05d005e52a1b8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:18:07.849137 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0df094a8-3416-4acd-9395-708713bd1948 service swift] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:07.849419 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0df094a8-3416-4acd-9395-708713bd1948 service swift] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:07.849689 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0df094a8-3416-4acd-9395-708713bd1948 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:07.859719 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0df094a8-3416-4acd-9395-708713bd1948 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:18:07.861301 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0df094a8-3416-4acd-9395-708713bd1948 service swift] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:18:07.870088 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1240/2511] 10.222.0.77 () {60 vars in 1384 bytes} [Tue Aug 30 05:18:07 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 141 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 30 05:18:17.866375 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=10 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:18:26.517625 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23235fcd-aae2-4e40-8d05-da456936f93a None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:18:26.527740 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23235fcd-aae2-4e40-8d05-da456936f93a None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:18:26.531283 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97c36de7-f49c-439a-a895-d2ce253915f5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:18:26.531866 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c36de7-f49c-439a-a895-d2ce253915f5 service neutron] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:26.532140 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c36de7-f49c-439a-a895-d2ce253915f5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:26.532405 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c36de7-f49c-439a-a895-d2ce253915f5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:26.534107 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-97c36de7-f49c-439a-a895-d2ce253915f5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:18:26.545570 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97c36de7-f49c-439a-a895-d2ce253915f5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:18:26.547389 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97c36de7-f49c-439a-a895-d2ce253915f5 service neutron] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:18:26.556992 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1272/2512] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:18:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 05:18:27.431172 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec75d22b-37b1-4202-9601-81b9a497ded6 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:27.431749 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec75d22b-37b1-4202-9601-81b9a497ded6 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:27.431749 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec75d22b-37b1-4202-9601-81b9a497ded6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:27.452074 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-ec75d22b-37b1-4202-9601-81b9a497ded6 None None] MFA Rules not processed for user `fe51dbec91624ed1ab00b441e29a3dd5`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:18:27.454899 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-ec75d22b-37b1-4202-9601-81b9a497ded6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:18:27.460765 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1241/2513] 10.222.0.77 () {58 vars in 967 bytes} [Tue Aug 30 05:18:27 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:18:27.473986 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97c36de7-f49c-439a-a895-d2ce253915f5 service neutron] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:18:27.486085 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97c36de7-f49c-439a-a895-d2ce253915f5 service neutron] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:18:27.491799 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7dc0f8c-54c1-4443-97dd-26c9eabf1e7b service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a7f7afc3d44ec788c05d005e52a1b8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:18:27.492573 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7dc0f8c-54c1-4443-97dd-26c9eabf1e7b service swift] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:27.492856 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7dc0f8c-54c1-4443-97dd-26c9eabf1e7b service swift] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:27.493126 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7dc0f8c-54c1-4443-97dd-26c9eabf1e7b service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:27.502184 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7dc0f8c-54c1-4443-97dd-26c9eabf1e7b service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:18:27.503818 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7dc0f8c-54c1-4443-97dd-26c9eabf1e7b service swift] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:18:27.513182 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1273/2514] 10.222.0.77 () {60 vars in 1384 bytes} [Tue Aug 30 05:18:27 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 30 05:18:31.580403 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec75d22b-37b1-4202-9601-81b9a497ded6 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:18:31.581858 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-ec75d22b-37b1-4202-9601-81b9a497ded6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:18:31.608514 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec75d22b-37b1-4202-9601-81b9a497ded6 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:18:31.613151 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0468af7-05a0-4208-a43b-da30a58d25ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:18:31.613792 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0468af7-05a0-4208-a43b-da30a58d25ab admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:31.614067 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0468af7-05a0-4208-a43b-da30a58d25ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:31.614335 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0468af7-05a0-4208-a43b-da30a58d25ab admin admin] PATH_INFO: `/v3/users/a8f01d0042c84767899dc2b71bca4e12` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:31.616054 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0468af7-05a0-4208-a43b-da30a58d25ab admin admin] RBAC: Authorizing `identity:delete_user(user_id=a8f01d0042c84767899dc2b71bca4e12)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:18:31.618145 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0468af7-05a0-4208-a43b-da30a58d25ab admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:18:31.674681 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-a0468af7-05a0-4208-a43b-da30a58d25ab admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a8f01d0042c84767899dc2b71bca4e12'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:18:31.675643 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-a0468af7-05a0-4208-a43b-da30a58d25ab admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a8f01d0042c84767899dc2b71bca4e12'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:18:31.676337 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-a0468af7-05a0-4208-a43b-da30a58d25ab admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a8f01d0042c84767899dc2b71bca4e12'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:18:31.696443 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-a0468af7-05a0-4208-a43b-da30a58d25ab admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a8f01d0042c84767899dc2b71bca4e12'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:18:31.706187 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-a0468af7-05a0-4208-a43b-da30a58d25ab admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a8f01d0042c84767899dc2b71bca4e12'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:18:31.721134 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1242/2515] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:18:31 2022] DELETE /identity/v3/users/a8f01d0042c84767899dc2b71bca4e12 => generated 0 bytes in 141 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:18:31.894290 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7dc0f8c-54c1-4443-97dd-26c9eabf1e7b service swift] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:18:31.905789 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7dc0f8c-54c1-4443-97dd-26c9eabf1e7b service swift] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:18:31.965252 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6a250ca-caf3-4db0-8f83-a70586a2f9bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:18:31.966126 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6a250ca-caf3-4db0-8f83-a70586a2f9bf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:31.966400 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6a250ca-caf3-4db0-8f83-a70586a2f9bf admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:31.966665 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6a250ca-caf3-4db0-8f83-a70586a2f9bf admin admin] PATH_INFO: `/v3/projects/116fb506b7794c1fa9460b5a861de136` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:31.967776 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6a250ca-caf3-4db0-8f83-a70586a2f9bf admin admin] RBAC: Authorizing `identity:delete_project(project_id=116fb506b7794c1fa9460b5a861de136)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:18:31.969661 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6a250ca-caf3-4db0-8f83-a70586a2f9bf admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:18:32.015916 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-f6a250ca-caf3-4db0-8f83-a70586a2f9bf admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '116fb506b7794c1fa9460b5a861de136', 'request_id': 'req-f6a250ca-caf3-4db0-8f83-a70586a2f9bf'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:18:32.023510 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-f6a250ca-caf3-4db0-8f83-a70586a2f9bf admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '116fb506b7794c1fa9460b5a861de136', 'request_id': 'req-f6a250ca-caf3-4db0-8f83-a70586a2f9bf'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:18:32.030501 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-f6a250ca-caf3-4db0-8f83-a70586a2f9bf admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '116fb506b7794c1fa9460b5a861de136', 'request_id': 'req-f6a250ca-caf3-4db0-8f83-a70586a2f9bf'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:18:32.041237 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-f6a250ca-caf3-4db0-8f83-a70586a2f9bf admin admin] The token cache is being invalidate because project 116fb506b7794c1fa9460b5a861de136 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81625) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:18:32.041555 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-f6a250ca-caf3-4db0-8f83-a70586a2f9bf admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:18:32.042192 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-f6a250ca-caf3-4db0-8f83-a70586a2f9bf admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:18:32.043724 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1274/2516] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:18:31 2022] DELETE /identity/v3/projects/116fb506b7794c1fa9460b5a861de136 => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:18:32.048579 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c766bc1c-539e-4fa8-99a9-e5c1e2c637cc None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:32.048865 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c766bc1c-539e-4fa8-99a9-e5c1e2c637cc None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:32.049138 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c766bc1c-539e-4fa8-99a9-e5c1e2c637cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:32.068942 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-c766bc1c-539e-4fa8-99a9-e5c1e2c637cc None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:18:32.070136 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-c766bc1c-539e-4fa8-99a9-e5c1e2c637cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:18:32.128148 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1243/2517] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:18:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:18:32.136579 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a0f6660-66ba-48f4-b629-417eecafcd44 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:32.136868 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a0f6660-66ba-48f4-b629-417eecafcd44 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:32.137143 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a0f6660-66ba-48f4-b629-417eecafcd44 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:32.156443 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-6a0f6660-66ba-48f4-b629-417eecafcd44 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:18:32.157649 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-6a0f6660-66ba-48f4-b629-417eecafcd44 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:18:32.165524 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1275/2518] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:18:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:18:32.169461 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c766bc1c-539e-4fa8-99a9-e5c1e2c637cc None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:18:32.179545 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c766bc1c-539e-4fa8-99a9-e5c1e2c637cc None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:18:32.184187 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-396feaa4-69fc-4075-a01b-2682f4d21429 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:18:32.184774 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-396feaa4-69fc-4075-a01b-2682f4d21429 admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:32.185051 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-396feaa4-69fc-4075-a01b-2682f4d21429 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:32.185321 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-396feaa4-69fc-4075-a01b-2682f4d21429 admin admin] PATH_INFO: `/v3/domains` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:32.185955 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-396feaa4-69fc-4075-a01b-2682f4d21429 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:18:32.186254 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-396feaa4-69fc-4075-a01b-2682f4d21429 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:18:32.187877 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-396feaa4-69fc-4075-a01b-2682f4d21429 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:18:32.199687 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1244/2519] 10.222.0.77 () {58 vars in 1153 bytes} [Tue Aug 30 05:18:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:18:32.203784 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a0f6660-66ba-48f4-b629-417eecafcd44 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:18:32.213372 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a0f6660-66ba-48f4-b629-417eecafcd44 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:18:32.218312 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6744a04-467e-45d2-8755-f5b43787a392 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:18:32.218935 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6744a04-467e-45d2-8755-f5b43787a392 admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:32.219216 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6744a04-467e-45d2-8755-f5b43787a392 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:32.219488 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6744a04-467e-45d2-8755-f5b43787a392 admin admin] PATH_INFO: `/v3/projects` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:32.220117 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6744a04-467e-45d2-8755-f5b43787a392 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:18:32.222002 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6744a04-467e-45d2-8755-f5b43787a392 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:18:32.248883 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1276/2520] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:18:32 2022] POST /identity/v3/projects => generated 383 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:18:32.252914 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-396feaa4-69fc-4075-a01b-2682f4d21429 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:18:32.262568 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-396feaa4-69fc-4075-a01b-2682f4d21429 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:18:32.309339 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:18:32.319396 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0e4d373-ce20-450c-9dd2-858123a123eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:18:32.320004 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e4d373-ce20-450c-9dd2-858123a123eb admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:32.320284 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e4d373-ce20-450c-9dd2-858123a123eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:32.320554 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e4d373-ce20-450c-9dd2-858123a123eb admin admin] PATH_INFO: `/v3/users` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:32.321179 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0e4d373-ce20-450c-9dd2-858123a123eb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:18:32.323155 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0e4d373-ce20-450c-9dd2-858123a123eb admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:18:32.345351 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1245/2521] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:18:32 2022] POST /identity/v3/users => generated 292 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:18:32.350068 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6744a04-467e-45d2-8755-f5b43787a392 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:18:32.360289 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6744a04-467e-45d2-8755-f5b43787a392 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:18:32.365129 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8468c0c-6efd-4f63-bc09-f4f865f61510 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:18:32.365747 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8468c0c-6efd-4f63-bc09-f4f865f61510 admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:32.366027 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8468c0c-6efd-4f63-bc09-f4f865f61510 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:32.366298 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8468c0c-6efd-4f63-bc09-f4f865f61510 admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:32.366970 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8468c0c-6efd-4f63-bc09-f4f865f61510 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:18:32.368552 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8468c0c-6efd-4f63-bc09-f4f865f61510 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:18:32.378270 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1277/2522] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:18:32 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:18:32.381971 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0e4d373-ce20-450c-9dd2-858123a123eb admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:18:32.391811 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0e4d373-ce20-450c-9dd2-858123a123eb admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:18:32.397020 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90fbb0f2-40f7-44f7-bd8e-97c6e44eaa87 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:18:32.397598 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90fbb0f2-40f7-44f7-bd8e-97c6e44eaa87 admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:32.397877 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90fbb0f2-40f7-44f7-bd8e-97c6e44eaa87 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:32.398143 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90fbb0f2-40f7-44f7-bd8e-97c6e44eaa87 admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:32.399013 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90fbb0f2-40f7-44f7-bd8e-97c6e44eaa87 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:18:32.400598 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90fbb0f2-40f7-44f7-bd8e-97c6e44eaa87 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:18:32.411000 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1246/2523] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:18:32 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:18:32.415589 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8468c0c-6efd-4f63-bc09-f4f865f61510 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:18:32.425193 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8468c0c-6efd-4f63-bc09-f4f865f61510 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:18:32.430042 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d0910b6-0fbb-45e8-b600-15398579d2d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:18:32.430679 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d0910b6-0fbb-45e8-b600-15398579d2d7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:32.430999 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d0910b6-0fbb-45e8-b600-15398579d2d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:32.431271 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d0910b6-0fbb-45e8-b600-15398579d2d7 admin admin] PATH_INFO: `/v3/projects/4ab62e21e615415ebe36b51bd9055c79/users/3360e6177b2d4626b9d3148196761829/roles/6f03d86980c34c36b44a8d6764ff4fda` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:32.445844 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d0910b6-0fbb-45e8-b600-15398579d2d7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4ab62e21e615415ebe36b51bd9055c79, user_id=3360e6177b2d4626b9d3148196761829, role_id=6f03d86980c34c36b44a8d6764ff4fda)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:18:32.448533 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d0910b6-0fbb-45e8-b600-15398579d2d7 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:18:32.457609 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:18:32.468320 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1278/2524] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:18:32 2022] PUT /identity/v3/projects/4ab62e21e615415ebe36b51bd9055c79/users/3360e6177b2d4626b9d3148196761829/roles/6f03d86980c34c36b44a8d6764ff4fda => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:18:32.479750 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83923e28-0018-4905-9dbb-58e24df99c80 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:32.480034 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83923e28-0018-4905-9dbb-58e24df99c80 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:32.480303 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83923e28-0018-4905-9dbb-58e24df99c80 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:32.508017 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-83923e28-0018-4905-9dbb-58e24df99c80 None None] MFA Rules not processed for user `3360e6177b2d4626b9d3148196761829`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:18:32.509272 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-83923e28-0018-4905-9dbb-58e24df99c80 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:18:32.574204 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1247/2525] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:18:32 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:18:32.585721 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d0910b6-0fbb-45e8-b600-15398579d2d7 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:18:32.587441 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-5d0910b6-0fbb-45e8-b600-15398579d2d7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:18:32.658940 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d0910b6-0fbb-45e8-b600-15398579d2d7 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:18:32.664412 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd17a479-c8f7-4c6c-a0ae-a85111ecc39f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '886eb5e00bbf4ec292eb962b665b37d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:18:32.665005 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd17a479-c8f7-4c6c-a0ae-a85111ecc39f service nova] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:32.665280 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd17a479-c8f7-4c6c-a0ae-a85111ecc39f service nova] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:32.665548 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd17a479-c8f7-4c6c-a0ae-a85111ecc39f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:32.673026 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd17a479-c8f7-4c6c-a0ae-a85111ecc39f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:18:32.674582 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd17a479-c8f7-4c6c-a0ae-a85111ecc39f service nova] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:18:32.683405 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1279/2526] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:18:32 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:18:39.963336 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83923e28-0018-4905-9dbb-58e24df99c80 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:18:40.028636 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83923e28-0018-4905-9dbb-58e24df99c80 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:18:40.033073 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e7f0b8c-a5cb-4263-a366-80a3117efcbf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:18:40.033689 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e7f0b8c-a5cb-4263-a366-80a3117efcbf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:40.033965 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e7f0b8c-a5cb-4263-a366-80a3117efcbf admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:40.034233 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e7f0b8c-a5cb-4263-a366-80a3117efcbf admin admin] PATH_INFO: `/v3/users/3360e6177b2d4626b9d3148196761829` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:40.035339 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e7f0b8c-a5cb-4263-a366-80a3117efcbf admin admin] RBAC: Authorizing `identity:delete_user(user_id=3360e6177b2d4626b9d3148196761829)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:18:40.037752 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e7f0b8c-a5cb-4263-a366-80a3117efcbf admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:18:40.080764 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-9e7f0b8c-a5cb-4263-a366-80a3117efcbf admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3360e6177b2d4626b9d3148196761829'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:18:40.081363 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-9e7f0b8c-a5cb-4263-a366-80a3117efcbf admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3360e6177b2d4626b9d3148196761829'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:18:40.081969 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-9e7f0b8c-a5cb-4263-a366-80a3117efcbf admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3360e6177b2d4626b9d3148196761829'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:18:40.092945 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=88,keystone:INSERT=3,keystone:DELETE=9 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:18:40.094285 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-9e7f0b8c-a5cb-4263-a366-80a3117efcbf admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3360e6177b2d4626b9d3148196761829'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:18:40.102381 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-9e7f0b8c-a5cb-4263-a366-80a3117efcbf admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3360e6177b2d4626b9d3148196761829'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:18:40.115758 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1248/2527] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:18:39 2022] DELETE /identity/v3/users/3360e6177b2d4626b9d3148196761829 => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:18:40.120183 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d5a219-98b6-4623-9044-7983398b5b66 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:40.120639 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d5a219-98b6-4623-9044-7983398b5b66 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:40.120734 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d5a219-98b6-4623-9044-7983398b5b66 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:40.139066 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-05d5a219-98b6-4623-9044-7983398b5b66 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:18:40.140244 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-05d5a219-98b6-4623-9044-7983398b5b66 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:18:40.196993 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1280/2528] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:18:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:18:40.206490 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e7f0b8c-a5cb-4263-a366-80a3117efcbf admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:18:40.207909 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-9e7f0b8c-a5cb-4263-a366-80a3117efcbf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:18:40.265523 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e7f0b8c-a5cb-4263-a366-80a3117efcbf admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:18:40.269103 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af9da026-ee53-419b-812e-6db444a4476f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:18:40.269690 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af9da026-ee53-419b-812e-6db444a4476f service neutron] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:40.269966 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af9da026-ee53-419b-812e-6db444a4476f service neutron] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:40.270234 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af9da026-ee53-419b-812e-6db444a4476f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:40.279537 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af9da026-ee53-419b-812e-6db444a4476f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:18:40.280939 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af9da026-ee53-419b-812e-6db444a4476f service neutron] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:18:40.290141 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1249/2529] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:18:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:18:40.549358 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05d5a219-98b6-4623-9044-7983398b5b66 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:18:40.550773 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-05d5a219-98b6-4623-9044-7983398b5b66 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:18:40.563029 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05d5a219-98b6-4623-9044-7983398b5b66 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:18:40.568231 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-765cf16f-720d-4c66-86f9-10c55dd5fcf9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:18:40.568863 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-765cf16f-720d-4c66-86f9-10c55dd5fcf9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:40.569140 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-765cf16f-720d-4c66-86f9-10c55dd5fcf9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:40.569406 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-765cf16f-720d-4c66-86f9-10c55dd5fcf9 admin admin] PATH_INFO: `/v3/projects/4ab62e21e615415ebe36b51bd9055c79` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:40.570512 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-765cf16f-720d-4c66-86f9-10c55dd5fcf9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4ab62e21e615415ebe36b51bd9055c79)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:18:40.572433 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-765cf16f-720d-4c66-86f9-10c55dd5fcf9 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:18:40.620042 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-765cf16f-720d-4c66-86f9-10c55dd5fcf9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4ab62e21e615415ebe36b51bd9055c79', 'request_id': 'req-765cf16f-720d-4c66-86f9-10c55dd5fcf9'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:18:40.627179 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-765cf16f-720d-4c66-86f9-10c55dd5fcf9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4ab62e21e615415ebe36b51bd9055c79', 'request_id': 'req-765cf16f-720d-4c66-86f9-10c55dd5fcf9'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:18:40.633930 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-765cf16f-720d-4c66-86f9-10c55dd5fcf9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4ab62e21e615415ebe36b51bd9055c79', 'request_id': 'req-765cf16f-720d-4c66-86f9-10c55dd5fcf9'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:18:40.644121 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-765cf16f-720d-4c66-86f9-10c55dd5fcf9 admin admin] The token cache is being invalidate because project 4ab62e21e615415ebe36b51bd9055c79 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81625) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:18:40.644350 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-765cf16f-720d-4c66-86f9-10c55dd5fcf9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:18:40.644972 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-765cf16f-720d-4c66-86f9-10c55dd5fcf9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:18:40.646442 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1281/2530] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:18:40 2022] DELETE /identity/v3/projects/4ab62e21e615415ebe36b51bd9055c79 => generated 0 bytes in 98 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:18:40.652578 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab912f0d-e23a-412e-bb9a-b98a041fd105 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:40.652855 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab912f0d-e23a-412e-bb9a-b98a041fd105 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:40.653123 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab912f0d-e23a-412e-bb9a-b98a041fd105 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:40.672190 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-ab912f0d-e23a-412e-bb9a-b98a041fd105 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:18:40.673387 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-ab912f0d-e23a-412e-bb9a-b98a041fd105 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:18:40.731972 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1250/2531] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:18:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:18:40.740266 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30a1b53d-bce3-4ad2-80b4-ecca3c811f38 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:40.740654 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30a1b53d-bce3-4ad2-80b4-ecca3c811f38 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:40.740933 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30a1b53d-bce3-4ad2-80b4-ecca3c811f38 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:40.760781 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-30a1b53d-bce3-4ad2-80b4-ecca3c811f38 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:18:40.761973 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-30a1b53d-bce3-4ad2-80b4-ecca3c811f38 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:18:40.769748 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1282/2532] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:18:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:18:40.774644 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab912f0d-e23a-412e-bb9a-b98a041fd105 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:18:40.784128 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab912f0d-e23a-412e-bb9a-b98a041fd105 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:18:40.789057 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62f6972a-f7a8-42f0-bfad-4a39365fc292 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:18:40.789640 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62f6972a-f7a8-42f0-bfad-4a39365fc292 admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:40.789914 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62f6972a-f7a8-42f0-bfad-4a39365fc292 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:40.790180 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62f6972a-f7a8-42f0-bfad-4a39365fc292 admin admin] PATH_INFO: `/v3/domains` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:40.790805 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62f6972a-f7a8-42f0-bfad-4a39365fc292 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:18:40.791126 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62f6972a-f7a8-42f0-bfad-4a39365fc292 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:18:40.792689 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62f6972a-f7a8-42f0-bfad-4a39365fc292 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:18:40.804398 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1251/2533] 10.222.0.77 () {58 vars in 1153 bytes} [Tue Aug 30 05:18:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:18:40.808588 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30a1b53d-bce3-4ad2-80b4-ecca3c811f38 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:18:40.813081 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:18:40.819386 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30a1b53d-bce3-4ad2-80b4-ecca3c811f38 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:18:40.825112 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60283f7c-400a-4001-9875-19d596f7706e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:18:40.825698 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60283f7c-400a-4001-9875-19d596f7706e admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:40.825975 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60283f7c-400a-4001-9875-19d596f7706e admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:40.826245 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60283f7c-400a-4001-9875-19d596f7706e admin admin] PATH_INFO: `/v3/projects` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:40.826908 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60283f7c-400a-4001-9875-19d596f7706e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:18:40.828805 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60283f7c-400a-4001-9875-19d596f7706e admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:18:40.857215 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1283/2534] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:18:40 2022] POST /identity/v3/projects => generated 387 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:18:40.861349 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62f6972a-f7a8-42f0-bfad-4a39365fc292 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:18:40.871976 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62f6972a-f7a8-42f0-bfad-4a39365fc292 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:18:40.928383 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f70f17f8-4d00-44fe-a178-57d5145630c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:18:40.928904 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70f17f8-4d00-44fe-a178-57d5145630c6 admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:40.929201 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70f17f8-4d00-44fe-a178-57d5145630c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:40.929470 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70f17f8-4d00-44fe-a178-57d5145630c6 admin admin] PATH_INFO: `/v3/users` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:40.930095 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f70f17f8-4d00-44fe-a178-57d5145630c6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:18:40.932072 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f70f17f8-4d00-44fe-a178-57d5145630c6 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:18:40.948393 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=95,keystone:INSERT=4,keystone:DELETE=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:18:40.954172 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1252/2535] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:18:40 2022] POST /identity/v3/users => generated 294 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:18:40.958181 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60283f7c-400a-4001-9875-19d596f7706e admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:18:40.968756 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60283f7c-400a-4001-9875-19d596f7706e admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:18:40.973417 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50f1fdb7-6619-47f8-9cbc-5a2811d0b00b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:18:40.974554 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50f1fdb7-6619-47f8-9cbc-5a2811d0b00b admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:40.974888 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50f1fdb7-6619-47f8-9cbc-5a2811d0b00b admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:40.975164 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50f1fdb7-6619-47f8-9cbc-5a2811d0b00b admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:40.975798 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50f1fdb7-6619-47f8-9cbc-5a2811d0b00b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:18:40.977382 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50f1fdb7-6619-47f8-9cbc-5a2811d0b00b admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:18:40.987780 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1284/2536] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:18:40 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:18:40.991496 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f70f17f8-4d00-44fe-a178-57d5145630c6 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:18:41.001419 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f70f17f8-4d00-44fe-a178-57d5145630c6 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:18:41.006026 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8177d933-2403-4245-a015-f057ad3cd1ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:18:41.006613 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8177d933-2403-4245-a015-f057ad3cd1ca admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:41.006923 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8177d933-2403-4245-a015-f057ad3cd1ca admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:41.007196 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8177d933-2403-4245-a015-f057ad3cd1ca admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:41.007828 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8177d933-2403-4245-a015-f057ad3cd1ca admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:18:41.009399 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8177d933-2403-4245-a015-f057ad3cd1ca admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:18:41.020791 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1253/2537] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:18:40 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:18:41.024999 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50f1fdb7-6619-47f8-9cbc-5a2811d0b00b admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:18:41.034987 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50f1fdb7-6619-47f8-9cbc-5a2811d0b00b admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:18:41.040117 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-838195ba-02a9-4ccf-a292-aa60f6cd0d05 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:18:41.040756 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-838195ba-02a9-4ccf-a292-aa60f6cd0d05 admin admin] REQUEST_METHOD: `PUT` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:41.041036 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-838195ba-02a9-4ccf-a292-aa60f6cd0d05 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:41.041311 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-838195ba-02a9-4ccf-a292-aa60f6cd0d05 admin admin] PATH_INFO: `/v3/projects/72a40baad4a047c8b70d80208989d286/users/dc6300b0fdf442679479b39c60cf0687/roles/6f03d86980c34c36b44a8d6764ff4fda` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:41.056243 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-838195ba-02a9-4ccf-a292-aa60f6cd0d05 admin admin] RBAC: Authorizing `identity:create_grant(project_id=72a40baad4a047c8b70d80208989d286, user_id=dc6300b0fdf442679479b39c60cf0687, role_id=6f03d86980c34c36b44a8d6764ff4fda)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:18:41.058936 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-838195ba-02a9-4ccf-a292-aa60f6cd0d05 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:18:41.071540 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1285/2538] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:18:41 2022] PUT /identity/v3/projects/72a40baad4a047c8b70d80208989d286/users/dc6300b0fdf442679479b39c60cf0687/roles/6f03d86980c34c36b44a8d6764ff4fda => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:18:41.081773 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-963944a3-592a-4609-b92a-e537b569ecf3 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:41.082059 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-963944a3-592a-4609-b92a-e537b569ecf3 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:41.082328 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-963944a3-592a-4609-b92a-e537b569ecf3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:41.110508 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-963944a3-592a-4609-b92a-e537b569ecf3 None None] MFA Rules not processed for user `dc6300b0fdf442679479b39c60cf0687`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:18:41.111807 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-963944a3-592a-4609-b92a-e537b569ecf3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:18:41.175800 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1254/2539] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:18:41 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:18:41.180967 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-838195ba-02a9-4ccf-a292-aa60f6cd0d05 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:18:41.242616 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-838195ba-02a9-4ccf-a292-aa60f6cd0d05 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:18:41.247285 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec40e60a-90e4-4d09-92a3-79d63cd93dc9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:18:41.247879 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec40e60a-90e4-4d09-92a3-79d63cd93dc9 admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:41.248157 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec40e60a-90e4-4d09-92a3-79d63cd93dc9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:41.248428 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec40e60a-90e4-4d09-92a3-79d63cd93dc9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:41.249049 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec40e60a-90e4-4d09-92a3-79d63cd93dc9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:18:41.250988 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec40e60a-90e4-4d09-92a3-79d63cd93dc9 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:18:41.276960 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1286/2540] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:18:41 2022] POST /identity/v3/projects => generated 385 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:18:41.286316 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-963944a3-592a-4609-b92a-e537b569ecf3 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:18:41.297173 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-963944a3-592a-4609-b92a-e537b569ecf3 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:18:41.353070 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17d73ec0-93b9-4876-88d7-73c844cb3fe9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:18:41.353666 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d73ec0-93b9-4876-88d7-73c844cb3fe9 admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:41.353947 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d73ec0-93b9-4876-88d7-73c844cb3fe9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:41.354462 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d73ec0-93b9-4876-88d7-73c844cb3fe9 admin admin] PATH_INFO: `/v3/users` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:41.355301 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17d73ec0-93b9-4876-88d7-73c844cb3fe9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:18:41.357221 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17d73ec0-93b9-4876-88d7-73c844cb3fe9 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:18:41.378500 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1255/2541] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:18:41 2022] POST /identity/v3/users => generated 299 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:18:41.383122 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec40e60a-90e4-4d09-92a3-79d63cd93dc9 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:18:41.393285 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec40e60a-90e4-4d09-92a3-79d63cd93dc9 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:18:41.397944 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de9eeed9-1a86-4bdb-8646-9f21dd90939e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:18:41.398529 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de9eeed9-1a86-4bdb-8646-9f21dd90939e admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:41.398808 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de9eeed9-1a86-4bdb-8646-9f21dd90939e admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:41.399108 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de9eeed9-1a86-4bdb-8646-9f21dd90939e admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:41.399766 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de9eeed9-1a86-4bdb-8646-9f21dd90939e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:18:41.401384 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de9eeed9-1a86-4bdb-8646-9f21dd90939e admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:18:41.412622 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1287/2542] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:18:41 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:18:41.416511 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17d73ec0-93b9-4876-88d7-73c844cb3fe9 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:18:41.426198 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17d73ec0-93b9-4876-88d7-73c844cb3fe9 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:18:41.431331 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96a7e4f5-b1b5-4203-8c8b-1370e0be6558 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:18:41.431904 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96a7e4f5-b1b5-4203-8c8b-1370e0be6558 admin admin] REQUEST_METHOD: `PUT` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:41.432183 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96a7e4f5-b1b5-4203-8c8b-1370e0be6558 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:41.432455 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96a7e4f5-b1b5-4203-8c8b-1370e0be6558 admin admin] PATH_INFO: `/v3/projects/3e61f4557b974be7a28f87e5856e4fd7/users/7201646da8a94d72bfd1406ed013ff16/roles/9d9d5d2c270146488249b7ed943280db` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:41.447248 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96a7e4f5-b1b5-4203-8c8b-1370e0be6558 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3e61f4557b974be7a28f87e5856e4fd7, user_id=7201646da8a94d72bfd1406ed013ff16, role_id=9d9d5d2c270146488249b7ed943280db)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:18:41.449877 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96a7e4f5-b1b5-4203-8c8b-1370e0be6558 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:18:41.463502 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1256/2543] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:18:41 2022] PUT /identity/v3/projects/3e61f4557b974be7a28f87e5856e4fd7/users/7201646da8a94d72bfd1406ed013ff16/roles/9d9d5d2c270146488249b7ed943280db => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:18:41.467797 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de9eeed9-1a86-4bdb-8646-9f21dd90939e admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:18:41.478311 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de9eeed9-1a86-4bdb-8646-9f21dd90939e admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:18:41.534750 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-989ff809-70ef-4218-bc1d-b207171d45f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:18:41.535385 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-989ff809-70ef-4218-bc1d-b207171d45f4 admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:41.535655 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-989ff809-70ef-4218-bc1d-b207171d45f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:41.535920 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-989ff809-70ef-4218-bc1d-b207171d45f4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:41.537073 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-989ff809-70ef-4218-bc1d-b207171d45f4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:18:41.538626 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-989ff809-70ef-4218-bc1d-b207171d45f4 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:18:41.548322 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1288/2544] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:18:41 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:18:41.553259 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96a7e4f5-b1b5-4203-8c8b-1370e0be6558 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:18:41.563357 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96a7e4f5-b1b5-4203-8c8b-1370e0be6558 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:18:41.567885 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3445a862-e563-4cde-bcd3-d103fb294202 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:18:41.568518 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3445a862-e563-4cde-bcd3-d103fb294202 admin admin] REQUEST_METHOD: `PUT` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:41.568795 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3445a862-e563-4cde-bcd3-d103fb294202 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:41.569063 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3445a862-e563-4cde-bcd3-d103fb294202 admin admin] PATH_INFO: `/v3/projects/3e61f4557b974be7a28f87e5856e4fd7/users/7201646da8a94d72bfd1406ed013ff16/roles/9d9d5d2c270146488249b7ed943280db` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:41.571099 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3445a862-e563-4cde-bcd3-d103fb294202 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3e61f4557b974be7a28f87e5856e4fd7, user_id=7201646da8a94d72bfd1406ed013ff16, role_id=9d9d5d2c270146488249b7ed943280db)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:18:41.573755 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3445a862-e563-4cde-bcd3-d103fb294202 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:18:41.584184 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1257/2545] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:18:41 2022] PUT /identity/v3/projects/3e61f4557b974be7a28f87e5856e4fd7/users/7201646da8a94d72bfd1406ed013ff16/roles/9d9d5d2c270146488249b7ed943280db => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:18:41.596067 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61f6af2c-1a2f-4656-8e3a-2a274e33683c None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:41.596354 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61f6af2c-1a2f-4656-8e3a-2a274e33683c None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:41.596626 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61f6af2c-1a2f-4656-8e3a-2a274e33683c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:41.624761 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-61f6af2c-1a2f-4656-8e3a-2a274e33683c None None] MFA Rules not processed for user `7201646da8a94d72bfd1406ed013ff16`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:18:41.625967 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-61f6af2c-1a2f-4656-8e3a-2a274e33683c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:18:41.629959 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:18:41.694891 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1289/2546] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:18:41 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:18:41.705256 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3445a862-e563-4cde-bcd3-d103fb294202 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:18:41.706995 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-3445a862-e563-4cde-bcd3-d103fb294202 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:18:41.742606 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:18:41.768331 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3445a862-e563-4cde-bcd3-d103fb294202 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:18:41.771981 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2994aeb1-cd04-4a76-8e2d-93614c7b4be8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '019a79b2affe478b8a30d1c538d0415c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:18:41.772574 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2994aeb1-cd04-4a76-8e2d-93614c7b4be8 service cinder] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:41.772857 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2994aeb1-cd04-4a76-8e2d-93614c7b4be8 service cinder] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:41.773124 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2994aeb1-cd04-4a76-8e2d-93614c7b4be8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:41.781907 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2994aeb1-cd04-4a76-8e2d-93614c7b4be8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:18:41.783367 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2994aeb1-cd04-4a76-8e2d-93614c7b4be8 service cinder] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:18:41.793134 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1258/2547] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:18:41 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:18:42.352772 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61f6af2c-1a2f-4656-8e3a-2a274e33683c None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:18:42.413796 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61f6af2c-1a2f-4656-8e3a-2a274e33683c None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:18:42.418252 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7ee280c-a835-47b8-965d-3b7346b89161 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:18:42.418887 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7ee280c-a835-47b8-965d-3b7346b89161 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:42.419166 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7ee280c-a835-47b8-965d-3b7346b89161 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:42.419436 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7ee280c-a835-47b8-965d-3b7346b89161 admin admin] PATH_INFO: `/v3/users/dc6300b0fdf442679479b39c60cf0687` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:42.420532 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7ee280c-a835-47b8-965d-3b7346b89161 admin admin] RBAC: Authorizing `identity:delete_user(user_id=dc6300b0fdf442679479b39c60cf0687)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:18:42.422441 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7ee280c-a835-47b8-965d-3b7346b89161 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:18:42.468541 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-d7ee280c-a835-47b8-965d-3b7346b89161 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'dc6300b0fdf442679479b39c60cf0687'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:18:42.475815 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-d7ee280c-a835-47b8-965d-3b7346b89161 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'dc6300b0fdf442679479b39c60cf0687'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:18:42.487479 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-d7ee280c-a835-47b8-965d-3b7346b89161 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'dc6300b0fdf442679479b39c60cf0687'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:18:42.494155 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-d7ee280c-a835-47b8-965d-3b7346b89161 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'dc6300b0fdf442679479b39c60cf0687'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:18:42.494773 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-d7ee280c-a835-47b8-965d-3b7346b89161 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'dc6300b0fdf442679479b39c60cf0687'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:18:42.500501 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1290/2548] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:18:42 2022] DELETE /identity/v3/users/dc6300b0fdf442679479b39c60cf0687 => generated 0 bytes in 148 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:18:42.504914 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12fda5ea-1c16-4912-a5f0-4c02713b4e42 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:42.505866 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12fda5ea-1c16-4912-a5f0-4c02713b4e42 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:42.505981 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12fda5ea-1c16-4912-a5f0-4c02713b4e42 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:42.524707 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-12fda5ea-1c16-4912-a5f0-4c02713b4e42 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:18:42.525875 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-12fda5ea-1c16-4912-a5f0-4c02713b4e42 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:18:42.582061 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1259/2549] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:18:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:18:42.592179 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7ee280c-a835-47b8-965d-3b7346b89161 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:18:42.593564 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-d7ee280c-a835-47b8-965d-3b7346b89161 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:18:42.615633 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:18:42.651602 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7ee280c-a835-47b8-965d-3b7346b89161 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:18:42.655187 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0dc794c-628a-4925-be1c-39da26d69132 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:18:42.655775 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0dc794c-628a-4925-be1c-39da26d69132 service neutron] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:42.656046 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0dc794c-628a-4925-be1c-39da26d69132 service neutron] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:42.656316 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0dc794c-628a-4925-be1c-39da26d69132 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:42.665684 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0dc794c-628a-4925-be1c-39da26d69132 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:18:42.667123 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0dc794c-628a-4925-be1c-39da26d69132 service neutron] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:18:42.676253 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1291/2550] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:18:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:18:42.931015 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12fda5ea-1c16-4912-a5f0-4c02713b4e42 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:18:42.932882 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-12fda5ea-1c16-4912-a5f0-4c02713b4e42 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:18:42.949764 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12fda5ea-1c16-4912-a5f0-4c02713b4e42 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:18:42.954519 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a559390-d6af-44fa-aa3f-ab07210c9b8d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:18:42.955258 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a559390-d6af-44fa-aa3f-ab07210c9b8d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:42.955601 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a559390-d6af-44fa-aa3f-ab07210c9b8d admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:42.955877 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a559390-d6af-44fa-aa3f-ab07210c9b8d admin admin] PATH_INFO: `/v3/projects/72a40baad4a047c8b70d80208989d286` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:42.957051 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a559390-d6af-44fa-aa3f-ab07210c9b8d admin admin] RBAC: Authorizing `identity:delete_project(project_id=72a40baad4a047c8b70d80208989d286)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:18:42.959585 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a559390-d6af-44fa-aa3f-ab07210c9b8d admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:18:43.005609 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-3a559390-d6af-44fa-aa3f-ab07210c9b8d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '72a40baad4a047c8b70d80208989d286', 'request_id': 'req-3a559390-d6af-44fa-aa3f-ab07210c9b8d'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:18:43.014321 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-3a559390-d6af-44fa-aa3f-ab07210c9b8d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '72a40baad4a047c8b70d80208989d286', 'request_id': 'req-3a559390-d6af-44fa-aa3f-ab07210c9b8d'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:18:43.022005 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-3a559390-d6af-44fa-aa3f-ab07210c9b8d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '72a40baad4a047c8b70d80208989d286', 'request_id': 'req-3a559390-d6af-44fa-aa3f-ab07210c9b8d'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:18:43.032607 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-3a559390-d6af-44fa-aa3f-ab07210c9b8d admin admin] The token cache is being invalidate because project 72a40baad4a047c8b70d80208989d286 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81624) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:18:43.032891 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-3a559390-d6af-44fa-aa3f-ab07210c9b8d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:18:43.033539 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-3a559390-d6af-44fa-aa3f-ab07210c9b8d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:18:43.035094 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1260/2551] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:18:42 2022] DELETE /identity/v3/projects/72a40baad4a047c8b70d80208989d286 => generated 0 bytes in 105 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:18:43.039537 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0dc794c-628a-4925-be1c-39da26d69132 service neutron] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:18:43.048947 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0dc794c-628a-4925-be1c-39da26d69132 service neutron] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:18:43.105232 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a3c0757-4529-4e58-ac86-2d213fd9ed33 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:18:43.105843 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a3c0757-4529-4e58-ac86-2d213fd9ed33 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:43.106119 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a3c0757-4529-4e58-ac86-2d213fd9ed33 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:43.106384 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a3c0757-4529-4e58-ac86-2d213fd9ed33 admin admin] PATH_INFO: `/v3/users/7201646da8a94d72bfd1406ed013ff16` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:43.107492 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a3c0757-4529-4e58-ac86-2d213fd9ed33 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7201646da8a94d72bfd1406ed013ff16)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:18:43.109446 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a3c0757-4529-4e58-ac86-2d213fd9ed33 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:18:43.153838 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-9a3c0757-4529-4e58-ac86-2d213fd9ed33 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7201646da8a94d72bfd1406ed013ff16'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:18:43.160546 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-9a3c0757-4529-4e58-ac86-2d213fd9ed33 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7201646da8a94d72bfd1406ed013ff16'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:18:43.173195 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-9a3c0757-4529-4e58-ac86-2d213fd9ed33 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7201646da8a94d72bfd1406ed013ff16'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:18:43.180747 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-9a3c0757-4529-4e58-ac86-2d213fd9ed33 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7201646da8a94d72bfd1406ed013ff16'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:18:43.181383 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-9a3c0757-4529-4e58-ac86-2d213fd9ed33 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7201646da8a94d72bfd1406ed013ff16'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:18:43.187137 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1292/2552] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:18:43 2022] DELETE /identity/v3/users/7201646da8a94d72bfd1406ed013ff16 => generated 0 bytes in 148 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:18:43.459424 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a559390-d6af-44fa-aa3f-ab07210c9b8d admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:18:43.460988 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-3a559390-d6af-44fa-aa3f-ab07210c9b8d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:18:43.474615 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a559390-d6af-44fa-aa3f-ab07210c9b8d admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:18:43.524837 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:18:43.531230 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b76e798-8d48-4a38-9196-2d44581c16be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:18:43.532133 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b76e798-8d48-4a38-9196-2d44581c16be admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:43.532415 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b76e798-8d48-4a38-9196-2d44581c16be admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:43.532690 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b76e798-8d48-4a38-9196-2d44581c16be admin admin] PATH_INFO: `/v3/projects/3e61f4557b974be7a28f87e5856e4fd7` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:43.534058 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b76e798-8d48-4a38-9196-2d44581c16be admin admin] RBAC: Authorizing `identity:delete_project(project_id=3e61f4557b974be7a28f87e5856e4fd7)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:18:43.536937 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b76e798-8d48-4a38-9196-2d44581c16be admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:18:43.581010 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-5b76e798-8d48-4a38-9196-2d44581c16be admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3e61f4557b974be7a28f87e5856e4fd7', 'request_id': 'req-5b76e798-8d48-4a38-9196-2d44581c16be'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:18:43.588437 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-5b76e798-8d48-4a38-9196-2d44581c16be admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3e61f4557b974be7a28f87e5856e4fd7', 'request_id': 'req-5b76e798-8d48-4a38-9196-2d44581c16be'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:18:43.594720 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-5b76e798-8d48-4a38-9196-2d44581c16be admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3e61f4557b974be7a28f87e5856e4fd7', 'request_id': 'req-5b76e798-8d48-4a38-9196-2d44581c16be'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:18:43.605456 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-5b76e798-8d48-4a38-9196-2d44581c16be admin admin] The token cache is being invalidate because project 3e61f4557b974be7a28f87e5856e4fd7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81624) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:18:43.605742 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-5b76e798-8d48-4a38-9196-2d44581c16be admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:18:43.606362 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-5b76e798-8d48-4a38-9196-2d44581c16be admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:18:43.607847 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1261/2553] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:18:43 2022] DELETE /identity/v3/projects/3e61f4557b974be7a28f87e5856e4fd7 => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:18:43.612854 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9028dc53-5c4b-4a82-8ca8-ec502bce1caa None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:43.613322 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9028dc53-5c4b-4a82-8ca8-ec502bce1caa None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:43.613418 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9028dc53-5c4b-4a82-8ca8-ec502bce1caa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:43.633278 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-9028dc53-5c4b-4a82-8ca8-ec502bce1caa None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:18:43.634477 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-9028dc53-5c4b-4a82-8ca8-ec502bce1caa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:18:43.673540 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:18:43.695148 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1293/2554] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:18:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:18:43.703038 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5695d208-4558-40d6-bb07-9c2c47f10516 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:43.703325 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5695d208-4558-40d6-bb07-9c2c47f10516 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:43.703601 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5695d208-4558-40d6-bb07-9c2c47f10516 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:43.723620 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-5695d208-4558-40d6-bb07-9c2c47f10516 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:18:43.724810 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-5695d208-4558-40d6-bb07-9c2c47f10516 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:18:43.732544 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1262/2555] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:18:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:18:43.736638 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9028dc53-5c4b-4a82-8ca8-ec502bce1caa None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:18:43.746162 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9028dc53-5c4b-4a82-8ca8-ec502bce1caa None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:18:43.750708 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-463114fc-3610-4b1e-bea1-94c8b0c84e44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:18:43.751322 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-463114fc-3610-4b1e-bea1-94c8b0c84e44 admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:43.751600 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-463114fc-3610-4b1e-bea1-94c8b0c84e44 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:43.751867 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-463114fc-3610-4b1e-bea1-94c8b0c84e44 admin admin] PATH_INFO: `/v3/domains` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:43.752505 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-463114fc-3610-4b1e-bea1-94c8b0c84e44 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:18:43.752802 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-463114fc-3610-4b1e-bea1-94c8b0c84e44 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:18:43.754379 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-463114fc-3610-4b1e-bea1-94c8b0c84e44 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:18:43.766213 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1294/2556] 10.222.0.77 () {58 vars in 1153 bytes} [Tue Aug 30 05:18:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:18:43.770454 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5695d208-4558-40d6-bb07-9c2c47f10516 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:18:43.780612 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5695d208-4558-40d6-bb07-9c2c47f10516 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:18:43.785165 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-538556de-cd06-4b96-928a-6a26e76e1faa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:18:43.785748 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-538556de-cd06-4b96-928a-6a26e76e1faa admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:43.786019 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-538556de-cd06-4b96-928a-6a26e76e1faa admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:43.786833 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-538556de-cd06-4b96-928a-6a26e76e1faa admin admin] PATH_INFO: `/v3/projects` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:43.787503 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-538556de-cd06-4b96-928a-6a26e76e1faa admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:18:43.789331 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-538556de-cd06-4b96-928a-6a26e76e1faa admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:18:43.815366 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1263/2557] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:18:43 2022] POST /identity/v3/projects => generated 379 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:18:43.819528 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-463114fc-3610-4b1e-bea1-94c8b0c84e44 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:18:43.829494 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-463114fc-3610-4b1e-bea1-94c8b0c84e44 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:18:43.885883 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bfa7714-4080-4922-a46c-96fbfbf631e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:18:43.886468 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bfa7714-4080-4922-a46c-96fbfbf631e1 admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:43.886747 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bfa7714-4080-4922-a46c-96fbfbf631e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:43.887045 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bfa7714-4080-4922-a46c-96fbfbf631e1 admin admin] PATH_INFO: `/v3/users` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:43.887665 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bfa7714-4080-4922-a46c-96fbfbf631e1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:18:43.889586 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bfa7714-4080-4922-a46c-96fbfbf631e1 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:18:43.910090 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1295/2558] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:18:43 2022] POST /identity/v3/users => generated 290 bytes in 91 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:18:43.914703 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-538556de-cd06-4b96-928a-6a26e76e1faa admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:18:43.924975 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-538556de-cd06-4b96-928a-6a26e76e1faa admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:18:43.929993 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15f8405d-bb2d-498c-8024-6eaabf0f3214 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:18:43.930573 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15f8405d-bb2d-498c-8024-6eaabf0f3214 admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:43.930944 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15f8405d-bb2d-498c-8024-6eaabf0f3214 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:43.931213 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15f8405d-bb2d-498c-8024-6eaabf0f3214 admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:43.931834 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15f8405d-bb2d-498c-8024-6eaabf0f3214 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:18:43.933431 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15f8405d-bb2d-498c-8024-6eaabf0f3214 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:18:43.943087 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1264/2559] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:18:43 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:18:43.946608 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bfa7714-4080-4922-a46c-96fbfbf631e1 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:18:43.956272 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bfa7714-4080-4922-a46c-96fbfbf631e1 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:18:43.960982 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-043ea531-88d4-49e5-a9e7-ef1de786d957 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:18:43.961561 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-043ea531-88d4-49e5-a9e7-ef1de786d957 admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:43.961840 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-043ea531-88d4-49e5-a9e7-ef1de786d957 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:43.962104 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-043ea531-88d4-49e5-a9e7-ef1de786d957 admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:43.962729 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-043ea531-88d4-49e5-a9e7-ef1de786d957 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:18:43.964331 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-043ea531-88d4-49e5-a9e7-ef1de786d957 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:18:43.976563 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1296/2560] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:18:43 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:18:43.980647 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15f8405d-bb2d-498c-8024-6eaabf0f3214 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:18:43.990844 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15f8405d-bb2d-498c-8024-6eaabf0f3214 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:18:43.995361 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9573194f-0100-4ed4-a84f-955f869306bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:18:43.995981 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9573194f-0100-4ed4-a84f-955f869306bb admin admin] REQUEST_METHOD: `PUT` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:43.996253 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9573194f-0100-4ed4-a84f-955f869306bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:43.996523 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9573194f-0100-4ed4-a84f-955f869306bb admin admin] PATH_INFO: `/v3/projects/ba28b19289b8439596f8d3e877a40076/users/b013c2e2d7054b4bb8889f9800ec4fd5/roles/6f03d86980c34c36b44a8d6764ff4fda` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:44.010494 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9573194f-0100-4ed4-a84f-955f869306bb admin admin] RBAC: Authorizing `identity:create_grant(project_id=ba28b19289b8439596f8d3e877a40076, user_id=b013c2e2d7054b4bb8889f9800ec4fd5, role_id=6f03d86980c34c36b44a8d6764ff4fda)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:18:44.013145 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9573194f-0100-4ed4-a84f-955f869306bb admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:18:44.025974 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1265/2561] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:18:43 2022] PUT /identity/v3/projects/ba28b19289b8439596f8d3e877a40076/users/b013c2e2d7054b4bb8889f9800ec4fd5/roles/6f03d86980c34c36b44a8d6764ff4fda => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:18:44.037811 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c80a97b5-a178-4d1e-8c41-66ce09f578be None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:44.038027 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c80a97b5-a178-4d1e-8c41-66ce09f578be None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:44.038291 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c80a97b5-a178-4d1e-8c41-66ce09f578be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:44.065211 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-c80a97b5-a178-4d1e-8c41-66ce09f578be None None] MFA Rules not processed for user `b013c2e2d7054b4bb8889f9800ec4fd5`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:18:44.066424 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-c80a97b5-a178-4d1e-8c41-66ce09f578be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:18:44.128552 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1297/2562] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:18:44 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:18:44.132976 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9573194f-0100-4ed4-a84f-955f869306bb admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:18:44.193355 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9573194f-0100-4ed4-a84f-955f869306bb admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:18:44.197957 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80728d92-d75e-4e55-aeb5-bd7708c85da7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:18:44.198551 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80728d92-d75e-4e55-aeb5-bd7708c85da7 admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:44.198876 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80728d92-d75e-4e55-aeb5-bd7708c85da7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:44.199147 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80728d92-d75e-4e55-aeb5-bd7708c85da7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:44.199777 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80728d92-d75e-4e55-aeb5-bd7708c85da7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:18:44.201672 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80728d92-d75e-4e55-aeb5-bd7708c85da7 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:18:44.228881 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1266/2563] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:18:44 2022] POST /identity/v3/projects => generated 379 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:18:44.232912 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c80a97b5-a178-4d1e-8c41-66ce09f578be None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:18:44.243647 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c80a97b5-a178-4d1e-8c41-66ce09f578be None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:18:44.277438 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:18:44.303298 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e54753e-3d50-4575-8760-7be04ecd189a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:18:44.303901 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e54753e-3d50-4575-8760-7be04ecd189a admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:44.304211 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e54753e-3d50-4575-8760-7be04ecd189a admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:44.304508 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e54753e-3d50-4575-8760-7be04ecd189a admin admin] PATH_INFO: `/v3/users` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:44.305175 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e54753e-3d50-4575-8760-7be04ecd189a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:18:44.307109 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e54753e-3d50-4575-8760-7be04ecd189a admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:18:44.328284 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1298/2564] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:18:44 2022] POST /identity/v3/users => generated 296 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:18:44.332495 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80728d92-d75e-4e55-aeb5-bd7708c85da7 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:18:44.343605 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80728d92-d75e-4e55-aeb5-bd7708c85da7 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:18:44.348182 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcf262ad-f29e-4812-9e3b-8b9a1eef4040 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:18:44.349378 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcf262ad-f29e-4812-9e3b-8b9a1eef4040 admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:44.349663 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcf262ad-f29e-4812-9e3b-8b9a1eef4040 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:44.349932 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcf262ad-f29e-4812-9e3b-8b9a1eef4040 admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:44.350565 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcf262ad-f29e-4812-9e3b-8b9a1eef4040 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:18:44.352196 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcf262ad-f29e-4812-9e3b-8b9a1eef4040 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:18:44.354583 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:18:44.367774 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1267/2565] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:18:44 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 05:18:44.368963 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e54753e-3d50-4575-8760-7be04ecd189a admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:18:44.379481 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e54753e-3d50-4575-8760-7be04ecd189a admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:18:44.384082 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6a450c2-5d69-480b-a598-073d3a32cfa1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:18:44.384722 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a450c2-5d69-480b-a598-073d3a32cfa1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:44.384996 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a450c2-5d69-480b-a598-073d3a32cfa1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:44.385270 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a450c2-5d69-480b-a598-073d3a32cfa1 admin admin] PATH_INFO: `/v3/projects/3d910acab4c24a19b77c72c291f34507/users/2f65e28e92504ebd9debbf8e85933aed/roles/9d9d5d2c270146488249b7ed943280db` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:44.401266 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6a450c2-5d69-480b-a598-073d3a32cfa1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3d910acab4c24a19b77c72c291f34507, user_id=2f65e28e92504ebd9debbf8e85933aed, role_id=9d9d5d2c270146488249b7ed943280db)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:18:44.403957 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6a450c2-5d69-480b-a598-073d3a32cfa1 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:18:44.416807 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1299/2566] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:18:44 2022] PUT /identity/v3/projects/3d910acab4c24a19b77c72c291f34507/users/2f65e28e92504ebd9debbf8e85933aed/roles/9d9d5d2c270146488249b7ed943280db => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:18:44.420745 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcf262ad-f29e-4812-9e3b-8b9a1eef4040 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:18:44.430807 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcf262ad-f29e-4812-9e3b-8b9a1eef4040 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:18:44.489643 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0853d2a-1faa-4923-8972-6aeddeb56ae2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:18:44.490240 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0853d2a-1faa-4923-8972-6aeddeb56ae2 admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:44.490516 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0853d2a-1faa-4923-8972-6aeddeb56ae2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:44.490806 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0853d2a-1faa-4923-8972-6aeddeb56ae2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:44.491458 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0853d2a-1faa-4923-8972-6aeddeb56ae2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:18:44.493029 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0853d2a-1faa-4923-8972-6aeddeb56ae2 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:18:44.503694 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1268/2567] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:18:44 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:18:44.508429 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6a450c2-5d69-480b-a598-073d3a32cfa1 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:18:44.519547 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6a450c2-5d69-480b-a598-073d3a32cfa1 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:18:44.524744 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0a07a91-7a0d-42bc-b1e1-f5ed3e4a4944 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:18:44.525386 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0a07a91-7a0d-42bc-b1e1-f5ed3e4a4944 admin admin] REQUEST_METHOD: `PUT` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:44.525665 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0a07a91-7a0d-42bc-b1e1-f5ed3e4a4944 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:44.525938 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0a07a91-7a0d-42bc-b1e1-f5ed3e4a4944 admin admin] PATH_INFO: `/v3/projects/3d910acab4c24a19b77c72c291f34507/users/2f65e28e92504ebd9debbf8e85933aed/roles/9d9d5d2c270146488249b7ed943280db` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:44.527880 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0a07a91-7a0d-42bc-b1e1-f5ed3e4a4944 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3d910acab4c24a19b77c72c291f34507, user_id=2f65e28e92504ebd9debbf8e85933aed, role_id=9d9d5d2c270146488249b7ed943280db)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:18:44.530536 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0a07a91-7a0d-42bc-b1e1-f5ed3e4a4944 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:18:44.541572 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1300/2568] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:18:44 2022] PUT /identity/v3/projects/3d910acab4c24a19b77c72c291f34507/users/2f65e28e92504ebd9debbf8e85933aed/roles/9d9d5d2c270146488249b7ed943280db => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:18:44.551637 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feac5210-d142-4b28-bcfe-1546167f69ff None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:44.552299 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feac5210-d142-4b28-bcfe-1546167f69ff None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:44.552299 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feac5210-d142-4b28-bcfe-1546167f69ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:44.588978 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-feac5210-d142-4b28-bcfe-1546167f69ff None None] MFA Rules not processed for user `2f65e28e92504ebd9debbf8e85933aed`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:18:44.590130 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-feac5210-d142-4b28-bcfe-1546167f69ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:18:44.660295 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1269/2569] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:18:44 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:18:44.670919 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0a07a91-7a0d-42bc-b1e1-f5ed3e4a4944 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:18:44.672606 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-f0a07a91-7a0d-42bc-b1e1-f5ed3e4a4944 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:18:44.730587 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0a07a91-7a0d-42bc-b1e1-f5ed3e4a4944 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:18:44.734367 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76981275-4c5d-4006-a7d7-624b661bd538 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '019a79b2affe478b8a30d1c538d0415c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:18:44.734986 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76981275-4c5d-4006-a7d7-624b661bd538 service cinder] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:44.735269 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76981275-4c5d-4006-a7d7-624b661bd538 service cinder] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:44.735539 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76981275-4c5d-4006-a7d7-624b661bd538 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:44.744549 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76981275-4c5d-4006-a7d7-624b661bd538 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:18:44.746030 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76981275-4c5d-4006-a7d7-624b661bd538 service cinder] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:18:44.755946 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1301/2570] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:18:44 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:18:45.119989 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feac5210-d142-4b28-bcfe-1546167f69ff None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:18:45.181557 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feac5210-d142-4b28-bcfe-1546167f69ff None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:18:45.185965 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-707034c9-c23a-4772-b44b-c9a609d08b9d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:18:45.186577 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-707034c9-c23a-4772-b44b-c9a609d08b9d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:45.186875 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-707034c9-c23a-4772-b44b-c9a609d08b9d admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:45.187145 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-707034c9-c23a-4772-b44b-c9a609d08b9d admin admin] PATH_INFO: `/v3/users/b013c2e2d7054b4bb8889f9800ec4fd5` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:45.188231 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-707034c9-c23a-4772-b44b-c9a609d08b9d admin admin] RBAC: Authorizing `identity:delete_user(user_id=b013c2e2d7054b4bb8889f9800ec4fd5)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:18:45.190157 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-707034c9-c23a-4772-b44b-c9a609d08b9d admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:18:45.212211 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:18:45.235484 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-707034c9-c23a-4772-b44b-c9a609d08b9d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b013c2e2d7054b4bb8889f9800ec4fd5'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:18:45.236122 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-707034c9-c23a-4772-b44b-c9a609d08b9d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b013c2e2d7054b4bb8889f9800ec4fd5'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:18:45.236724 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-707034c9-c23a-4772-b44b-c9a609d08b9d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b013c2e2d7054b4bb8889f9800ec4fd5'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:18:45.256321 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-707034c9-c23a-4772-b44b-c9a609d08b9d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b013c2e2d7054b4bb8889f9800ec4fd5'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:18:45.263724 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-707034c9-c23a-4772-b44b-c9a609d08b9d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b013c2e2d7054b4bb8889f9800ec4fd5'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:18:45.276588 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1270/2571] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:18:45 2022] DELETE /identity/v3/users/b013c2e2d7054b4bb8889f9800ec4fd5 => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 05:18:45.281071 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3e63394-49c4-4fb4-963e-a18b6544f5f4 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:45.281354 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3e63394-49c4-4fb4-963e-a18b6544f5f4 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:45.281622 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3e63394-49c4-4fb4-963e-a18b6544f5f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:45.300181 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-e3e63394-49c4-4fb4-963e-a18b6544f5f4 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:18:45.301067 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-e3e63394-49c4-4fb4-963e-a18b6544f5f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:18:45.359671 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1302/2572] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:18:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:18:45.369875 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-707034c9-c23a-4772-b44b-c9a609d08b9d admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:18:45.371988 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-707034c9-c23a-4772-b44b-c9a609d08b9d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:18:45.431982 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-707034c9-c23a-4772-b44b-c9a609d08b9d admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:18:45.435660 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-804f20dc-3eac-4edf-af85-5e6750f58006 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:18:45.436254 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-804f20dc-3eac-4edf-af85-5e6750f58006 service neutron] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:45.436548 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-804f20dc-3eac-4edf-af85-5e6750f58006 service neutron] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:45.436805 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-804f20dc-3eac-4edf-af85-5e6750f58006 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:45.445586 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-804f20dc-3eac-4edf-af85-5e6750f58006 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:18:45.447036 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-804f20dc-3eac-4edf-af85-5e6750f58006 service neutron] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:18:45.456287 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1271/2573] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:18:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:18:45.713529 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3e63394-49c4-4fb4-963e-a18b6544f5f4 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:18:45.715071 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-e3e63394-49c4-4fb4-963e-a18b6544f5f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:18:45.727689 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3e63394-49c4-4fb4-963e-a18b6544f5f4 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:18:45.732220 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1521e9f4-a7cf-4e5a-a838-76b17268fedd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:18:45.732843 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1521e9f4-a7cf-4e5a-a838-76b17268fedd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:45.733122 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1521e9f4-a7cf-4e5a-a838-76b17268fedd admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:45.733391 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1521e9f4-a7cf-4e5a-a838-76b17268fedd admin admin] PATH_INFO: `/v3/projects/ba28b19289b8439596f8d3e877a40076` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:45.734503 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1521e9f4-a7cf-4e5a-a838-76b17268fedd admin admin] RBAC: Authorizing `identity:delete_project(project_id=ba28b19289b8439596f8d3e877a40076)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:18:45.736471 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1521e9f4-a7cf-4e5a-a838-76b17268fedd admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:18:45.769528 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=93,keystone:INSERT=5,keystone:DELETE=2 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:18:45.785913 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-1521e9f4-a7cf-4e5a-a838-76b17268fedd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ba28b19289b8439596f8d3e877a40076', 'request_id': 'req-1521e9f4-a7cf-4e5a-a838-76b17268fedd'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:18:45.793006 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-1521e9f4-a7cf-4e5a-a838-76b17268fedd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ba28b19289b8439596f8d3e877a40076', 'request_id': 'req-1521e9f4-a7cf-4e5a-a838-76b17268fedd'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:18:45.799234 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-1521e9f4-a7cf-4e5a-a838-76b17268fedd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ba28b19289b8439596f8d3e877a40076', 'request_id': 'req-1521e9f4-a7cf-4e5a-a838-76b17268fedd'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:18:45.809891 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-1521e9f4-a7cf-4e5a-a838-76b17268fedd admin admin] The token cache is being invalidate because project ba28b19289b8439596f8d3e877a40076 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81625) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:18:45.810175 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-1521e9f4-a7cf-4e5a-a838-76b17268fedd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:18:45.810794 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-1521e9f4-a7cf-4e5a-a838-76b17268fedd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:18:45.812272 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1303/2574] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:18:45 2022] DELETE /identity/v3/projects/ba28b19289b8439596f8d3e877a40076 => generated 0 bytes in 99 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:18:45.816245 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-804f20dc-3eac-4edf-af85-5e6750f58006 service neutron] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:18:45.826289 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-804f20dc-3eac-4edf-af85-5e6750f58006 service neutron] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:18:45.881692 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a83177dd-5387-4d66-83c2-6670d9bdac49 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:18:45.882307 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83177dd-5387-4d66-83c2-6670d9bdac49 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:45.882583 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83177dd-5387-4d66-83c2-6670d9bdac49 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:45.882926 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83177dd-5387-4d66-83c2-6670d9bdac49 admin admin] PATH_INFO: `/v3/users/2f65e28e92504ebd9debbf8e85933aed` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:45.884059 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a83177dd-5387-4d66-83c2-6670d9bdac49 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2f65e28e92504ebd9debbf8e85933aed)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:18:45.885967 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a83177dd-5387-4d66-83c2-6670d9bdac49 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:18:45.929543 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:DELETE=13,keystone:SELECT=86,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:18:45.932180 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-a83177dd-5387-4d66-83c2-6670d9bdac49 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2f65e28e92504ebd9debbf8e85933aed'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:18:45.933287 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-a83177dd-5387-4d66-83c2-6670d9bdac49 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2f65e28e92504ebd9debbf8e85933aed'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:18:45.933974 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-a83177dd-5387-4d66-83c2-6670d9bdac49 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2f65e28e92504ebd9debbf8e85933aed'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:18:45.946789 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-a83177dd-5387-4d66-83c2-6670d9bdac49 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2f65e28e92504ebd9debbf8e85933aed'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:18:45.953776 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-a83177dd-5387-4d66-83c2-6670d9bdac49 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2f65e28e92504ebd9debbf8e85933aed'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:18:45.967260 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1272/2575] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:18:45 2022] DELETE /identity/v3/users/2f65e28e92504ebd9debbf8e85933aed => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:18:46.230096 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1521e9f4-a7cf-4e5a-a838-76b17268fedd admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:18:46.231648 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-1521e9f4-a7cf-4e5a-a838-76b17268fedd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:18:46.245606 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1521e9f4-a7cf-4e5a-a838-76b17268fedd admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:18:46.303365 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29e1283a-b485-4dbc-8e2b-3d26ba1d3981 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:18:46.303993 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29e1283a-b485-4dbc-8e2b-3d26ba1d3981 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:46.304268 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29e1283a-b485-4dbc-8e2b-3d26ba1d3981 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:46.304534 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29e1283a-b485-4dbc-8e2b-3d26ba1d3981 admin admin] PATH_INFO: `/v3/projects/3d910acab4c24a19b77c72c291f34507` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:46.305632 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29e1283a-b485-4dbc-8e2b-3d26ba1d3981 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3d910acab4c24a19b77c72c291f34507)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:18:46.307576 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29e1283a-b485-4dbc-8e2b-3d26ba1d3981 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:18:46.353336 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-29e1283a-b485-4dbc-8e2b-3d26ba1d3981 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3d910acab4c24a19b77c72c291f34507', 'request_id': 'req-29e1283a-b485-4dbc-8e2b-3d26ba1d3981'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:18:46.359920 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-29e1283a-b485-4dbc-8e2b-3d26ba1d3981 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3d910acab4c24a19b77c72c291f34507', 'request_id': 'req-29e1283a-b485-4dbc-8e2b-3d26ba1d3981'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:18:46.366550 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-29e1283a-b485-4dbc-8e2b-3d26ba1d3981 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3d910acab4c24a19b77c72c291f34507', 'request_id': 'req-29e1283a-b485-4dbc-8e2b-3d26ba1d3981'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:18:46.376642 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-29e1283a-b485-4dbc-8e2b-3d26ba1d3981 admin admin] The token cache is being invalidate because project 3d910acab4c24a19b77c72c291f34507 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81625) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:18:46.376932 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-29e1283a-b485-4dbc-8e2b-3d26ba1d3981 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:18:46.377554 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-29e1283a-b485-4dbc-8e2b-3d26ba1d3981 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:18:46.379029 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1304/2576] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:18:46 2022] DELETE /identity/v3/projects/3d910acab4c24a19b77c72c291f34507 => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:18:46.384397 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b38c584a-0ce9-4e22-adc4-a7cf194803ce None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:46.384686 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b38c584a-0ce9-4e22-adc4-a7cf194803ce None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:46.384964 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b38c584a-0ce9-4e22-adc4-a7cf194803ce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:46.404252 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-b38c584a-0ce9-4e22-adc4-a7cf194803ce None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:18:46.405428 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-b38c584a-0ce9-4e22-adc4-a7cf194803ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:18:46.462776 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1273/2577] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:18:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:18:46.470921 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc416816-041c-4636-9eff-a43cc4a2f388 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:46.471210 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc416816-041c-4636-9eff-a43cc4a2f388 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:46.471483 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc416816-041c-4636-9eff-a43cc4a2f388 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:46.492184 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-cc416816-041c-4636-9eff-a43cc4a2f388 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:18:46.493378 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-cc416816-041c-4636-9eff-a43cc4a2f388 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:18:46.501195 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1305/2578] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:18:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:18:46.505893 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b38c584a-0ce9-4e22-adc4-a7cf194803ce None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:18:46.515688 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b38c584a-0ce9-4e22-adc4-a7cf194803ce None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:18:46.520268 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44519410-29a6-4b20-8c2f-be3cf2e85ded admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:18:46.520860 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44519410-29a6-4b20-8c2f-be3cf2e85ded admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:46.521140 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44519410-29a6-4b20-8c2f-be3cf2e85ded admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:46.521441 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44519410-29a6-4b20-8c2f-be3cf2e85ded admin admin] PATH_INFO: `/v3/domains` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:46.522077 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44519410-29a6-4b20-8c2f-be3cf2e85ded admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:18:46.522404 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44519410-29a6-4b20-8c2f-be3cf2e85ded admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:18:46.524047 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44519410-29a6-4b20-8c2f-be3cf2e85ded admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:18:46.535269 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1274/2579] 10.222.0.77 () {58 vars in 1153 bytes} [Tue Aug 30 05:18:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:18:46.539329 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc416816-041c-4636-9eff-a43cc4a2f388 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:18:46.548482 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc416816-041c-4636-9eff-a43cc4a2f388 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:18:46.553560 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-096d26e3-48e1-4dbc-a3c9-746058054854 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:18:46.554134 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096d26e3-48e1-4dbc-a3c9-746058054854 admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:46.554410 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096d26e3-48e1-4dbc-a3c9-746058054854 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:46.554676 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096d26e3-48e1-4dbc-a3c9-746058054854 admin admin] PATH_INFO: `/v3/projects` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:46.555336 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-096d26e3-48e1-4dbc-a3c9-746058054854 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:18:46.557216 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-096d26e3-48e1-4dbc-a3c9-746058054854 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:18:46.583277 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1306/2580] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:18:46 2022] POST /identity/v3/projects => generated 377 bytes in 44 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:18:46.587260 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44519410-29a6-4b20-8c2f-be3cf2e85ded admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:18:46.597524 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44519410-29a6-4b20-8c2f-be3cf2e85ded admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:18:46.653006 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fafcf2dc-43d6-48e4-b96b-291115d24bf4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:18:46.653601 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fafcf2dc-43d6-48e4-b96b-291115d24bf4 admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:46.653885 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fafcf2dc-43d6-48e4-b96b-291115d24bf4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:46.654172 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fafcf2dc-43d6-48e4-b96b-291115d24bf4 admin admin] PATH_INFO: `/v3/users` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:46.654786 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fafcf2dc-43d6-48e4-b96b-291115d24bf4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:18:46.656746 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fafcf2dc-43d6-48e4-b96b-291115d24bf4 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:18:46.679249 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1275/2581] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:18:46 2022] POST /identity/v3/users => generated 289 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:18:46.683351 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-096d26e3-48e1-4dbc-a3c9-746058054854 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:18:46.694298 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-096d26e3-48e1-4dbc-a3c9-746058054854 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:18:46.698867 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5beb8c45-8d34-4d11-8a40-22f80cb32ccc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:18:46.699459 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5beb8c45-8d34-4d11-8a40-22f80cb32ccc admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:46.699759 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5beb8c45-8d34-4d11-8a40-22f80cb32ccc admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:46.700009 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5beb8c45-8d34-4d11-8a40-22f80cb32ccc admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:46.700644 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5beb8c45-8d34-4d11-8a40-22f80cb32ccc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:18:46.702208 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5beb8c45-8d34-4d11-8a40-22f80cb32ccc admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:18:46.712334 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1307/2582] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:18:46 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:18:46.716236 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fafcf2dc-43d6-48e4-b96b-291115d24bf4 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:18:46.726434 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fafcf2dc-43d6-48e4-b96b-291115d24bf4 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:18:46.731026 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c967093-ce22-4fcb-bec3-01a1add7deca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:18:46.731598 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c967093-ce22-4fcb-bec3-01a1add7deca admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:46.731875 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c967093-ce22-4fcb-bec3-01a1add7deca admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:46.732143 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c967093-ce22-4fcb-bec3-01a1add7deca admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:46.732772 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c967093-ce22-4fcb-bec3-01a1add7deca admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:18:46.734345 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c967093-ce22-4fcb-bec3-01a1add7deca admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:18:46.745310 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1276/2583] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:18:46 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:18:46.749160 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5beb8c45-8d34-4d11-8a40-22f80cb32ccc admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:18:46.758999 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5beb8c45-8d34-4d11-8a40-22f80cb32ccc admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:18:46.763715 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d860d726-0d4e-4447-b2d1-02772f483f0b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:18:46.764347 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d860d726-0d4e-4447-b2d1-02772f483f0b admin admin] REQUEST_METHOD: `PUT` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:46.764628 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d860d726-0d4e-4447-b2d1-02772f483f0b admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:46.764896 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d860d726-0d4e-4447-b2d1-02772f483f0b admin admin] PATH_INFO: `/v3/projects/7ac6097dd5fd4db09a11372e100ad57d/users/6c507e30d5714f06bf3b7c75755214e6/roles/6f03d86980c34c36b44a8d6764ff4fda` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:46.767188 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=85,keystone:DELETE=12,keystone:INSERT=3 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:18:46.781432 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d860d726-0d4e-4447-b2d1-02772f483f0b admin admin] RBAC: Authorizing `identity:create_grant(project_id=7ac6097dd5fd4db09a11372e100ad57d, user_id=6c507e30d5714f06bf3b7c75755214e6, role_id=6f03d86980c34c36b44a8d6764ff4fda)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:18:46.784101 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d860d726-0d4e-4447-b2d1-02772f483f0b admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:18:46.796664 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1308/2584] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:18:46 2022] PUT /identity/v3/projects/7ac6097dd5fd4db09a11372e100ad57d/users/6c507e30d5714f06bf3b7c75755214e6/roles/6f03d86980c34c36b44a8d6764ff4fda => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 05:18:46.808370 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b110c26-9d6f-456d-8d89-228293b799a4 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:46.808651 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b110c26-9d6f-456d-8d89-228293b799a4 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:46.808921 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b110c26-9d6f-456d-8d89-228293b799a4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:46.835651 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-0b110c26-9d6f-456d-8d89-228293b799a4 None None] MFA Rules not processed for user `6c507e30d5714f06bf3b7c75755214e6`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:18:46.836843 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-0b110c26-9d6f-456d-8d89-228293b799a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:18:46.849790 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:DELETE=4,keystone:SELECT=92,keystone:INSERT=4 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:18:46.903280 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1277/2585] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:18:46 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:18:46.907693 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d860d726-0d4e-4447-b2d1-02772f483f0b admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:18:46.967882 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d860d726-0d4e-4447-b2d1-02772f483f0b admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:18:46.972333 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f250d0ae-01d0-4948-8e90-ed338ebb1e2c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:18:46.972924 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f250d0ae-01d0-4948-8e90-ed338ebb1e2c admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:46.973203 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f250d0ae-01d0-4948-8e90-ed338ebb1e2c admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:46.973473 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f250d0ae-01d0-4948-8e90-ed338ebb1e2c admin admin] PATH_INFO: `/v3/projects` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:46.974602 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f250d0ae-01d0-4948-8e90-ed338ebb1e2c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:18:46.976543 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f250d0ae-01d0-4948-8e90-ed338ebb1e2c admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:18:47.002284 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1309/2586] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:18:46 2022] POST /identity/v3/projects => generated 377 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:18:47.006209 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b110c26-9d6f-456d-8d89-228293b799a4 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:18:47.017026 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b110c26-9d6f-456d-8d89-228293b799a4 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:18:47.073718 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eccbbdb-629d-45ef-83f8-8279a3c320d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:18:47.074314 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eccbbdb-629d-45ef-83f8-8279a3c320d7 admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:47.074588 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eccbbdb-629d-45ef-83f8-8279a3c320d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:47.074878 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eccbbdb-629d-45ef-83f8-8279a3c320d7 admin admin] PATH_INFO: `/v3/users` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:47.075498 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eccbbdb-629d-45ef-83f8-8279a3c320d7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:18:47.077399 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eccbbdb-629d-45ef-83f8-8279a3c320d7 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:18:47.097770 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1278/2587] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:18:47 2022] POST /identity/v3/users => generated 289 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:18:47.102342 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f250d0ae-01d0-4948-8e90-ed338ebb1e2c admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:18:47.112512 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f250d0ae-01d0-4948-8e90-ed338ebb1e2c admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:18:47.117191 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c1edfd3-93bc-47fb-8c73-9149f7af07c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:18:47.117770 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c1edfd3-93bc-47fb-8c73-9149f7af07c3 admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:47.118046 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c1edfd3-93bc-47fb-8c73-9149f7af07c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:47.118314 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c1edfd3-93bc-47fb-8c73-9149f7af07c3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:47.118979 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c1edfd3-93bc-47fb-8c73-9149f7af07c3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:18:47.120581 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c1edfd3-93bc-47fb-8c73-9149f7af07c3 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:18:47.135006 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1310/2588] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:18:47 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:18:47.138888 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eccbbdb-629d-45ef-83f8-8279a3c320d7 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:18:47.149518 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eccbbdb-629d-45ef-83f8-8279a3c320d7 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:18:47.154173 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-665ed6e1-9f58-4a38-9d4f-26008ee7f53c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:18:47.154762 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-665ed6e1-9f58-4a38-9d4f-26008ee7f53c admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:47.155118 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-665ed6e1-9f58-4a38-9d4f-26008ee7f53c admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:47.155389 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-665ed6e1-9f58-4a38-9d4f-26008ee7f53c admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:47.156018 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-665ed6e1-9f58-4a38-9d4f-26008ee7f53c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:18:47.157619 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-665ed6e1-9f58-4a38-9d4f-26008ee7f53c admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:18:47.168097 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1279/2589] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:18:47 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:18:47.172082 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c1edfd3-93bc-47fb-8c73-9149f7af07c3 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:18:47.181881 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c1edfd3-93bc-47fb-8c73-9149f7af07c3 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:18:47.186594 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd75fc7c-afdb-48e5-b849-d31591c8631b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:18:47.187254 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd75fc7c-afdb-48e5-b849-d31591c8631b admin admin] REQUEST_METHOD: `PUT` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:47.187532 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd75fc7c-afdb-48e5-b849-d31591c8631b admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:47.187803 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd75fc7c-afdb-48e5-b849-d31591c8631b admin admin] PATH_INFO: `/v3/projects/d32bc9bcadd74cfea80bc89b82a1667e/users/4f73aa5623034014bbb363d7d4eaa55c/roles/6f03d86980c34c36b44a8d6764ff4fda` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:47.202050 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd75fc7c-afdb-48e5-b849-d31591c8631b admin admin] RBAC: Authorizing `identity:create_grant(project_id=d32bc9bcadd74cfea80bc89b82a1667e, user_id=4f73aa5623034014bbb363d7d4eaa55c, role_id=6f03d86980c34c36b44a8d6764ff4fda)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:18:47.204709 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd75fc7c-afdb-48e5-b849-d31591c8631b admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:18:47.218036 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1311/2590] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:18:47 2022] PUT /identity/v3/projects/d32bc9bcadd74cfea80bc89b82a1667e/users/4f73aa5623034014bbb363d7d4eaa55c/roles/6f03d86980c34c36b44a8d6764ff4fda => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 05:18:47.228859 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4af154e-6c25-4332-9956-1e8e08751254 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:47.229139 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4af154e-6c25-4332-9956-1e8e08751254 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:47.229405 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4af154e-6c25-4332-9956-1e8e08751254 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:47.256662 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-b4af154e-6c25-4332-9956-1e8e08751254 None None] MFA Rules not processed for user `4f73aa5623034014bbb363d7d4eaa55c`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:18:47.257861 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-b4af154e-6c25-4332-9956-1e8e08751254 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:18:47.322296 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1280/2591] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:18:47 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:18:47.326701 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd75fc7c-afdb-48e5-b849-d31591c8631b admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:18:47.391953 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd75fc7c-afdb-48e5-b849-d31591c8631b admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:18:47.396845 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-226dd419-f927-4c11-8c44-012fd17e460c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:18:47.397448 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-226dd419-f927-4c11-8c44-012fd17e460c admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:47.397725 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-226dd419-f927-4c11-8c44-012fd17e460c admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:47.397992 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-226dd419-f927-4c11-8c44-012fd17e460c admin admin] PATH_INFO: `/v3/projects` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:47.398617 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-226dd419-f927-4c11-8c44-012fd17e460c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:18:47.400533 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-226dd419-f927-4c11-8c44-012fd17e460c admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:18:47.426093 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1312/2592] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:18:47 2022] POST /identity/v3/projects => generated 377 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:18:47.430639 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4af154e-6c25-4332-9956-1e8e08751254 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:18:47.441332 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4af154e-6c25-4332-9956-1e8e08751254 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:18:47.461704 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:18:47.500689 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0edc3bd5-9b2e-4c77-908b-7ebdaffcdf16 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:18:47.501218 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0edc3bd5-9b2e-4c77-908b-7ebdaffcdf16 admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:47.501491 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0edc3bd5-9b2e-4c77-908b-7ebdaffcdf16 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:47.501756 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0edc3bd5-9b2e-4c77-908b-7ebdaffcdf16 admin admin] PATH_INFO: `/v3/users` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:47.502377 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0edc3bd5-9b2e-4c77-908b-7ebdaffcdf16 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:18:47.504304 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0edc3bd5-9b2e-4c77-908b-7ebdaffcdf16 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:18:47.524945 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1281/2593] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:18:47 2022] POST /identity/v3/users => generated 295 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:18:47.528970 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-226dd419-f927-4c11-8c44-012fd17e460c admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:18:47.539741 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-226dd419-f927-4c11-8c44-012fd17e460c admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:18:47.544302 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd63cf35-a415-4842-ad22-6f3651f14d00 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:18:47.544887 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd63cf35-a415-4842-ad22-6f3651f14d00 admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:47.545167 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd63cf35-a415-4842-ad22-6f3651f14d00 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:47.545434 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd63cf35-a415-4842-ad22-6f3651f14d00 admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:47.546063 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd63cf35-a415-4842-ad22-6f3651f14d00 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:18:47.547881 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd63cf35-a415-4842-ad22-6f3651f14d00 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:18:47.560082 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1313/2594] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:18:47 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:18:47.564092 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0edc3bd5-9b2e-4c77-908b-7ebdaffcdf16 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:18:47.574347 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0edc3bd5-9b2e-4c77-908b-7ebdaffcdf16 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:18:47.579052 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-609900fa-3ab0-4d86-add0-bc9d02f003fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:18:47.579690 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-609900fa-3ab0-4d86-add0-bc9d02f003fe admin admin] REQUEST_METHOD: `PUT` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:47.579972 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-609900fa-3ab0-4d86-add0-bc9d02f003fe admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:47.580247 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-609900fa-3ab0-4d86-add0-bc9d02f003fe admin admin] PATH_INFO: `/v3/projects/b4dcd22423a246efaa1b839202384617/users/7203397ac04a4419a2edff56628638db/roles/9d9d5d2c270146488249b7ed943280db` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:47.594935 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-609900fa-3ab0-4d86-add0-bc9d02f003fe admin admin] RBAC: Authorizing `identity:create_grant(project_id=b4dcd22423a246efaa1b839202384617, user_id=7203397ac04a4419a2edff56628638db, role_id=9d9d5d2c270146488249b7ed943280db)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:18:47.597593 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-609900fa-3ab0-4d86-add0-bc9d02f003fe admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:18:47.610458 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1282/2595] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:18:47 2022] PUT /identity/v3/projects/b4dcd22423a246efaa1b839202384617/users/7203397ac04a4419a2edff56628638db/roles/9d9d5d2c270146488249b7ed943280db => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:18:47.614967 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd63cf35-a415-4842-ad22-6f3651f14d00 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:18:47.625069 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd63cf35-a415-4842-ad22-6f3651f14d00 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:18:47.628731 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:18:47.682995 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5372602-4847-40f7-a3f0-6415d53a6c42 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:18:47.683583 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5372602-4847-40f7-a3f0-6415d53a6c42 admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:47.683861 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5372602-4847-40f7-a3f0-6415d53a6c42 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:47.684130 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5372602-4847-40f7-a3f0-6415d53a6c42 admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:47.684753 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5372602-4847-40f7-a3f0-6415d53a6c42 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:18:47.686334 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5372602-4847-40f7-a3f0-6415d53a6c42 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:18:47.695809 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1314/2596] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:18:47 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:18:47.699875 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-609900fa-3ab0-4d86-add0-bc9d02f003fe admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:18:47.710105 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-609900fa-3ab0-4d86-add0-bc9d02f003fe admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:18:47.714811 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-191130d3-c7b6-4e88-8d18-d2686340fb43 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:18:47.715476 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-191130d3-c7b6-4e88-8d18-d2686340fb43 admin admin] REQUEST_METHOD: `PUT` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:47.716005 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-191130d3-c7b6-4e88-8d18-d2686340fb43 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:47.716272 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-191130d3-c7b6-4e88-8d18-d2686340fb43 admin admin] PATH_INFO: `/v3/projects/b4dcd22423a246efaa1b839202384617/users/7203397ac04a4419a2edff56628638db/roles/9d9d5d2c270146488249b7ed943280db` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:47.718163 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-191130d3-c7b6-4e88-8d18-d2686340fb43 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b4dcd22423a246efaa1b839202384617, user_id=7203397ac04a4419a2edff56628638db, role_id=9d9d5d2c270146488249b7ed943280db)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:18:47.720823 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-191130d3-c7b6-4e88-8d18-d2686340fb43 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:18:47.731299 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1283/2597] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:18:47 2022] PUT /identity/v3/projects/b4dcd22423a246efaa1b839202384617/users/7203397ac04a4419a2edff56628638db/roles/9d9d5d2c270146488249b7ed943280db => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:18:47.741731 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2de05397-f01d-44a2-aebe-38a2ef1df9ba None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:47.742019 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2de05397-f01d-44a2-aebe-38a2ef1df9ba None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:47.742319 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2de05397-f01d-44a2-aebe-38a2ef1df9ba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:47.769340 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-2de05397-f01d-44a2-aebe-38a2ef1df9ba None None] MFA Rules not processed for user `7203397ac04a4419a2edff56628638db`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:18:47.770479 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-2de05397-f01d-44a2-aebe-38a2ef1df9ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:18:47.837894 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1315/2598] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:18:47 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:18:47.850401 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-191130d3-c7b6-4e88-8d18-d2686340fb43 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:18:47.852146 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-191130d3-c7b6-4e88-8d18-d2686340fb43 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:18:47.907015 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-191130d3-c7b6-4e88-8d18-d2686340fb43 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:18:47.910566 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7acc23de-2f5f-4506-ba12-c15bddf4c12d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '019a79b2affe478b8a30d1c538d0415c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:18:47.911757 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7acc23de-2f5f-4506-ba12-c15bddf4c12d service cinder] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:47.912033 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7acc23de-2f5f-4506-ba12-c15bddf4c12d service cinder] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:47.912303 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7acc23de-2f5f-4506-ba12-c15bddf4c12d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:47.965482 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7acc23de-2f5f-4506-ba12-c15bddf4c12d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:18:47.967197 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7acc23de-2f5f-4506-ba12-c15bddf4c12d service cinder] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:18:47.976386 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1284/2599] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:18:47 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 126 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:18:49.471411 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2de05397-f01d-44a2-aebe-38a2ef1df9ba None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:18:49.472830 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-2de05397-f01d-44a2-aebe-38a2ef1df9ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:18:49.485284 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2de05397-f01d-44a2-aebe-38a2ef1df9ba None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:18:49.488951 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b456fafa-9924-4a06-a4ad-6e9d1bba6e7e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '019a79b2affe478b8a30d1c538d0415c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:18:49.489536 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b456fafa-9924-4a06-a4ad-6e9d1bba6e7e service cinder] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:49.489812 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b456fafa-9924-4a06-a4ad-6e9d1bba6e7e service cinder] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:49.490082 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b456fafa-9924-4a06-a4ad-6e9d1bba6e7e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:49.545816 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b456fafa-9924-4a06-a4ad-6e9d1bba6e7e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:18:49.547273 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b456fafa-9924-4a06-a4ad-6e9d1bba6e7e service cinder] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:18:49.555980 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1316/2600] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:18:49 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:18:50.056588 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7acc23de-2f5f-4506-ba12-c15bddf4c12d service cinder] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:18:50.065163 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7acc23de-2f5f-4506-ba12-c15bddf4c12d service cinder] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:18:50.069331 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef0bf51f-6779-465e-9bde-d2aa9f717398 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '019a79b2affe478b8a30d1c538d0415c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:18:50.069843 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef0bf51f-6779-465e-9bde-d2aa9f717398 service cinder] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:50.070122 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef0bf51f-6779-465e-9bde-d2aa9f717398 service cinder] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:50.070396 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef0bf51f-6779-465e-9bde-d2aa9f717398 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:50.079201 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef0bf51f-6779-465e-9bde-d2aa9f717398 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:18:50.080624 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef0bf51f-6779-465e-9bde-d2aa9f717398 service cinder] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:18:50.090803 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1285/2601] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:18:50 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 35 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:18:53.928427 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b456fafa-9924-4a06-a4ad-6e9d1bba6e7e service cinder] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:18:53.958768 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:18:53.991756 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b456fafa-9924-4a06-a4ad-6e9d1bba6e7e service cinder] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:18:53.996252 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa955869-5acd-44ea-a842-5bec9ff57b4b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:18:53.996863 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa955869-5acd-44ea-a842-5bec9ff57b4b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:53.997142 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa955869-5acd-44ea-a842-5bec9ff57b4b admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:53.997415 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa955869-5acd-44ea-a842-5bec9ff57b4b admin admin] PATH_INFO: `/v3/users/6c507e30d5714f06bf3b7c75755214e6` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:53.998516 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa955869-5acd-44ea-a842-5bec9ff57b4b admin admin] RBAC: Authorizing `identity:delete_user(user_id=6c507e30d5714f06bf3b7c75755214e6)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:18:54.000470 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa955869-5acd-44ea-a842-5bec9ff57b4b admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:18:54.045093 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-aa955869-5acd-44ea-a842-5bec9ff57b4b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6c507e30d5714f06bf3b7c75755214e6'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:18:54.052841 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-aa955869-5acd-44ea-a842-5bec9ff57b4b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6c507e30d5714f06bf3b7c75755214e6'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:18:54.064703 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-aa955869-5acd-44ea-a842-5bec9ff57b4b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6c507e30d5714f06bf3b7c75755214e6'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:18:54.072582 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-aa955869-5acd-44ea-a842-5bec9ff57b4b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6c507e30d5714f06bf3b7c75755214e6'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:18:54.073220 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-aa955869-5acd-44ea-a842-5bec9ff57b4b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6c507e30d5714f06bf3b7c75755214e6'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:18:54.079013 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1317/2602] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:18:53 2022] DELETE /identity/v3/users/6c507e30d5714f06bf3b7c75755214e6 => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:18:54.083613 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5bde67f-9d2a-4ee0-ab1f-6dcf477872d1 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:54.084112 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5bde67f-9d2a-4ee0-ab1f-6dcf477872d1 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:54.084215 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5bde67f-9d2a-4ee0-ab1f-6dcf477872d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:54.103030 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-b5bde67f-9d2a-4ee0-ab1f-6dcf477872d1 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:18:54.104233 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-b5bde67f-9d2a-4ee0-ab1f-6dcf477872d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:18:54.116352 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:18:54.164716 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1286/2603] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:18:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:18:54.174492 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa955869-5acd-44ea-a842-5bec9ff57b4b admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:18:54.175902 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-aa955869-5acd-44ea-a842-5bec9ff57b4b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:18:54.232737 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa955869-5acd-44ea-a842-5bec9ff57b4b admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:18:54.236349 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7101c573-f766-4790-a90e-29d05b59400d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:18:54.236940 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7101c573-f766-4790-a90e-29d05b59400d service neutron] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:54.237220 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7101c573-f766-4790-a90e-29d05b59400d service neutron] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:54.237491 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7101c573-f766-4790-a90e-29d05b59400d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:54.246315 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7101c573-f766-4790-a90e-29d05b59400d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:18:54.247756 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7101c573-f766-4790-a90e-29d05b59400d service neutron] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:18:54.257784 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1318/2604] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:18:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:18:54.536452 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5bde67f-9d2a-4ee0-ab1f-6dcf477872d1 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:18:54.538023 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-b5bde67f-9d2a-4ee0-ab1f-6dcf477872d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:18:54.550086 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5bde67f-9d2a-4ee0-ab1f-6dcf477872d1 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:18:54.565635 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8646b220-6fea-4c95-b845-9d9dd2de46a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:18:54.566282 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8646b220-6fea-4c95-b845-9d9dd2de46a5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:54.566561 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8646b220-6fea-4c95-b845-9d9dd2de46a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:54.566838 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8646b220-6fea-4c95-b845-9d9dd2de46a5 admin admin] PATH_INFO: `/v3/projects/7ac6097dd5fd4db09a11372e100ad57d` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:54.568512 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8646b220-6fea-4c95-b845-9d9dd2de46a5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7ac6097dd5fd4db09a11372e100ad57d)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:18:54.570295 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8646b220-6fea-4c95-b845-9d9dd2de46a5 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:18:54.617591 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-8646b220-6fea-4c95-b845-9d9dd2de46a5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7ac6097dd5fd4db09a11372e100ad57d', 'request_id': 'req-8646b220-6fea-4c95-b845-9d9dd2de46a5'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:18:54.625293 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-8646b220-6fea-4c95-b845-9d9dd2de46a5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7ac6097dd5fd4db09a11372e100ad57d', 'request_id': 'req-8646b220-6fea-4c95-b845-9d9dd2de46a5'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:18:54.632353 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-8646b220-6fea-4c95-b845-9d9dd2de46a5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7ac6097dd5fd4db09a11372e100ad57d', 'request_id': 'req-8646b220-6fea-4c95-b845-9d9dd2de46a5'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:18:54.642524 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-8646b220-6fea-4c95-b845-9d9dd2de46a5 admin admin] The token cache is being invalidate because project 7ac6097dd5fd4db09a11372e100ad57d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81624) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:18:54.642811 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-8646b220-6fea-4c95-b845-9d9dd2de46a5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:18:54.644070 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-8646b220-6fea-4c95-b845-9d9dd2de46a5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:18:54.646304 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1287/2605] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:18:54 2022] DELETE /identity/v3/projects/7ac6097dd5fd4db09a11372e100ad57d => generated 0 bytes in 111 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:18:54.650273 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7101c573-f766-4790-a90e-29d05b59400d service neutron] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:18:54.660432 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7101c573-f766-4790-a90e-29d05b59400d service neutron] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:18:54.705588 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:18:54.720191 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6a8c49f-1bc9-4c9b-8692-b1b5ebf39dcc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:18:54.720815 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6a8c49f-1bc9-4c9b-8692-b1b5ebf39dcc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:54.721089 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6a8c49f-1bc9-4c9b-8692-b1b5ebf39dcc admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:54.721364 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6a8c49f-1bc9-4c9b-8692-b1b5ebf39dcc admin admin] PATH_INFO: `/v3/users/4f73aa5623034014bbb363d7d4eaa55c` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:54.722494 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6a8c49f-1bc9-4c9b-8692-b1b5ebf39dcc admin admin] RBAC: Authorizing `identity:delete_user(user_id=4f73aa5623034014bbb363d7d4eaa55c)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:18:54.724501 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6a8c49f-1bc9-4c9b-8692-b1b5ebf39dcc admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:18:54.770764 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-d6a8c49f-1bc9-4c9b-8692-b1b5ebf39dcc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4f73aa5623034014bbb363d7d4eaa55c'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:18:54.777963 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-d6a8c49f-1bc9-4c9b-8692-b1b5ebf39dcc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4f73aa5623034014bbb363d7d4eaa55c'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:18:54.790001 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-d6a8c49f-1bc9-4c9b-8692-b1b5ebf39dcc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4f73aa5623034014bbb363d7d4eaa55c'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:18:54.796865 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-d6a8c49f-1bc9-4c9b-8692-b1b5ebf39dcc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4f73aa5623034014bbb363d7d4eaa55c'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:18:54.797505 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-d6a8c49f-1bc9-4c9b-8692-b1b5ebf39dcc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4f73aa5623034014bbb363d7d4eaa55c'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:18:54.803795 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1319/2606] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:18:54 2022] DELETE /identity/v3/users/4f73aa5623034014bbb363d7d4eaa55c => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:18:55.071054 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8646b220-6fea-4c95-b845-9d9dd2de46a5 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:18:55.072639 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-8646b220-6fea-4c95-b845-9d9dd2de46a5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:18:55.086438 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8646b220-6fea-4c95-b845-9d9dd2de46a5 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:18:55.144123 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1401da5a-62bd-45ec-bc09-d3fcf49e27f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:18:55.144744 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1401da5a-62bd-45ec-bc09-d3fcf49e27f4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:55.145024 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1401da5a-62bd-45ec-bc09-d3fcf49e27f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:55.145354 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1401da5a-62bd-45ec-bc09-d3fcf49e27f4 admin admin] PATH_INFO: `/v3/projects/d32bc9bcadd74cfea80bc89b82a1667e` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:55.146981 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1401da5a-62bd-45ec-bc09-d3fcf49e27f4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d32bc9bcadd74cfea80bc89b82a1667e)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:18:55.148905 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1401da5a-62bd-45ec-bc09-d3fcf49e27f4 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:18:55.190411 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:18:55.195252 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-1401da5a-62bd-45ec-bc09-d3fcf49e27f4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd32bc9bcadd74cfea80bc89b82a1667e', 'request_id': 'req-1401da5a-62bd-45ec-bc09-d3fcf49e27f4'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:18:55.203303 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-1401da5a-62bd-45ec-bc09-d3fcf49e27f4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd32bc9bcadd74cfea80bc89b82a1667e', 'request_id': 'req-1401da5a-62bd-45ec-bc09-d3fcf49e27f4'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:18:55.210086 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-1401da5a-62bd-45ec-bc09-d3fcf49e27f4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd32bc9bcadd74cfea80bc89b82a1667e', 'request_id': 'req-1401da5a-62bd-45ec-bc09-d3fcf49e27f4'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:18:55.220881 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-1401da5a-62bd-45ec-bc09-d3fcf49e27f4 admin admin] The token cache is being invalidate because project d32bc9bcadd74cfea80bc89b82a1667e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81624) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:18:55.221166 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-1401da5a-62bd-45ec-bc09-d3fcf49e27f4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:18:55.221816 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-1401da5a-62bd-45ec-bc09-d3fcf49e27f4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:18:55.223319 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1288/2607] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:18:55 2022] DELETE /identity/v3/projects/d32bc9bcadd74cfea80bc89b82a1667e => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:18:55.227698 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6a8c49f-1bc9-4c9b-8692-b1b5ebf39dcc admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:18:55.229097 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-d6a8c49f-1bc9-4c9b-8692-b1b5ebf39dcc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:18:55.242233 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6a8c49f-1bc9-4c9b-8692-b1b5ebf39dcc admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:18:55.299366 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aefe5d7-fa0f-46c5-b0ac-8e8da5d66d76 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:18:55.299980 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aefe5d7-fa0f-46c5-b0ac-8e8da5d66d76 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:55.300257 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aefe5d7-fa0f-46c5-b0ac-8e8da5d66d76 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:55.300522 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aefe5d7-fa0f-46c5-b0ac-8e8da5d66d76 admin admin] PATH_INFO: `/v3/users/7203397ac04a4419a2edff56628638db` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:55.302222 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aefe5d7-fa0f-46c5-b0ac-8e8da5d66d76 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7203397ac04a4419a2edff56628638db)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:18:55.304196 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aefe5d7-fa0f-46c5-b0ac-8e8da5d66d76 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:18:55.351059 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-9aefe5d7-fa0f-46c5-b0ac-8e8da5d66d76 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7203397ac04a4419a2edff56628638db'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:18:55.358603 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-9aefe5d7-fa0f-46c5-b0ac-8e8da5d66d76 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7203397ac04a4419a2edff56628638db'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:18:55.370273 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=78,keystone:DELETE=20,keystone:INSERT=2 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:18:55.371513 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-9aefe5d7-fa0f-46c5-b0ac-8e8da5d66d76 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7203397ac04a4419a2edff56628638db'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:18:55.380841 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-9aefe5d7-fa0f-46c5-b0ac-8e8da5d66d76 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7203397ac04a4419a2edff56628638db'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:18:55.381506 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-9aefe5d7-fa0f-46c5-b0ac-8e8da5d66d76 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7203397ac04a4419a2edff56628638db'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:18:55.387362 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1320/2608] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:18:55 2022] DELETE /identity/v3/users/7203397ac04a4419a2edff56628638db => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:18:55.647826 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1401da5a-62bd-45ec-bc09-d3fcf49e27f4 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:18:55.649272 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-1401da5a-62bd-45ec-bc09-d3fcf49e27f4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:18:55.663509 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1401da5a-62bd-45ec-bc09-d3fcf49e27f4 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:18:55.721655 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3174b870-e4d0-4826-957f-a9f4697414de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:18:55.722206 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3174b870-e4d0-4826-957f-a9f4697414de admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:55.722478 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3174b870-e4d0-4826-957f-a9f4697414de admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:55.722744 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3174b870-e4d0-4826-957f-a9f4697414de admin admin] PATH_INFO: `/v3/projects/b4dcd22423a246efaa1b839202384617` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:55.724374 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3174b870-e4d0-4826-957f-a9f4697414de admin admin] RBAC: Authorizing `identity:delete_project(project_id=b4dcd22423a246efaa1b839202384617)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:18:55.726257 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3174b870-e4d0-4826-957f-a9f4697414de admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:18:55.770599 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-3174b870-e4d0-4826-957f-a9f4697414de admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b4dcd22423a246efaa1b839202384617', 'request_id': 'req-3174b870-e4d0-4826-957f-a9f4697414de'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:18:55.777969 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-3174b870-e4d0-4826-957f-a9f4697414de admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b4dcd22423a246efaa1b839202384617', 'request_id': 'req-3174b870-e4d0-4826-957f-a9f4697414de'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:18:55.784221 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-3174b870-e4d0-4826-957f-a9f4697414de admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b4dcd22423a246efaa1b839202384617', 'request_id': 'req-3174b870-e4d0-4826-957f-a9f4697414de'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:18:55.794598 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-3174b870-e4d0-4826-957f-a9f4697414de admin admin] The token cache is being invalidate because project b4dcd22423a246efaa1b839202384617 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81624) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:18:55.794913 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-3174b870-e4d0-4826-957f-a9f4697414de admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:18:55.795521 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-3174b870-e4d0-4826-957f-a9f4697414de admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:18:55.796954 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1289/2609] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:18:55 2022] DELETE /identity/v3/projects/b4dcd22423a246efaa1b839202384617 => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:18:55.802403 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae8c7dd-19ce-4942-a020-e61b99225347 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:55.802691 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae8c7dd-19ce-4942-a020-e61b99225347 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:55.803000 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae8c7dd-19ce-4942-a020-e61b99225347 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:55.822258 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-fae8c7dd-19ce-4942-a020-e61b99225347 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:18:55.823490 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-fae8c7dd-19ce-4942-a020-e61b99225347 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:18:55.881555 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1321/2610] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:18:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:18:55.889317 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e52947a0-3b2a-4f4a-a22e-f1307812742e None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:55.889597 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e52947a0-3b2a-4f4a-a22e-f1307812742e None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:55.889864 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e52947a0-3b2a-4f4a-a22e-f1307812742e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:55.909364 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-e52947a0-3b2a-4f4a-a22e-f1307812742e None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:18:55.910570 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-e52947a0-3b2a-4f4a-a22e-f1307812742e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:18:55.919015 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1290/2611] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:18:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:18:55.922953 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fae8c7dd-19ce-4942-a020-e61b99225347 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:18:55.933132 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fae8c7dd-19ce-4942-a020-e61b99225347 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:18:55.937684 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03e9e958-a345-4878-b738-a7ef971d96a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:18:55.938272 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03e9e958-a345-4878-b738-a7ef971d96a2 admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:55.938549 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03e9e958-a345-4878-b738-a7ef971d96a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:55.938842 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03e9e958-a345-4878-b738-a7ef971d96a2 admin admin] PATH_INFO: `/v3/domains` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:55.939482 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03e9e958-a345-4878-b738-a7ef971d96a2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:18:55.939783 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03e9e958-a345-4878-b738-a7ef971d96a2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:18:55.941389 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03e9e958-a345-4878-b738-a7ef971d96a2 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:18:55.952704 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1322/2612] 10.222.0.77 () {58 vars in 1153 bytes} [Tue Aug 30 05:18:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:18:55.956996 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e52947a0-3b2a-4f4a-a22e-f1307812742e None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:18:55.966553 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e52947a0-3b2a-4f4a-a22e-f1307812742e None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:18:55.971264 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9ccfab2-4fc5-4421-b1ec-20cd5bc6fe41 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:18:55.971902 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ccfab2-4fc5-4421-b1ec-20cd5bc6fe41 admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:55.972123 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ccfab2-4fc5-4421-b1ec-20cd5bc6fe41 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:55.972389 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ccfab2-4fc5-4421-b1ec-20cd5bc6fe41 admin admin] PATH_INFO: `/v3/projects` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:55.973010 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9ccfab2-4fc5-4421-b1ec-20cd5bc6fe41 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:18:55.975430 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9ccfab2-4fc5-4421-b1ec-20cd5bc6fe41 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:18:56.001994 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1291/2613] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:18:55 2022] POST /identity/v3/projects => generated 365 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:18:56.006045 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03e9e958-a345-4878-b738-a7ef971d96a2 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:18:56.015862 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03e9e958-a345-4878-b738-a7ef971d96a2 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:18:56.072706 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-787268f7-8e24-49c6-b7b9-82818c1aa7d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:18:56.073375 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-787268f7-8e24-49c6-b7b9-82818c1aa7d1 admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:56.073655 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-787268f7-8e24-49c6-b7b9-82818c1aa7d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:56.073918 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-787268f7-8e24-49c6-b7b9-82818c1aa7d1 admin admin] PATH_INFO: `/v3/users` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:56.074523 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-787268f7-8e24-49c6-b7b9-82818c1aa7d1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:18:56.076433 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-787268f7-8e24-49c6-b7b9-82818c1aa7d1 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:18:56.097019 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1323/2614] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:18:56 2022] POST /identity/v3/users => generated 283 bytes in 91 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:18:56.101854 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9ccfab2-4fc5-4421-b1ec-20cd5bc6fe41 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:18:56.112031 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9ccfab2-4fc5-4421-b1ec-20cd5bc6fe41 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:18:56.116631 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-306e9539-1cae-4067-9b06-e897bc2d7e6a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:18:56.117214 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-306e9539-1cae-4067-9b06-e897bc2d7e6a admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:56.117493 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-306e9539-1cae-4067-9b06-e897bc2d7e6a admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:56.117761 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-306e9539-1cae-4067-9b06-e897bc2d7e6a admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:56.118390 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-306e9539-1cae-4067-9b06-e897bc2d7e6a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:18:56.120007 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-306e9539-1cae-4067-9b06-e897bc2d7e6a admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:18:56.129638 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1292/2615] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:18:56 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:18:56.133331 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-787268f7-8e24-49c6-b7b9-82818c1aa7d1 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:18:56.143161 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-787268f7-8e24-49c6-b7b9-82818c1aa7d1 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:18:56.148258 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-731aa30c-b0a5-4f81-a0d9-699893d31743 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:18:56.148822 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-731aa30c-b0a5-4f81-a0d9-699893d31743 admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:56.149097 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-731aa30c-b0a5-4f81-a0d9-699893d31743 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:56.149364 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-731aa30c-b0a5-4f81-a0d9-699893d31743 admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:56.149987 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-731aa30c-b0a5-4f81-a0d9-699893d31743 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:18:56.151588 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-731aa30c-b0a5-4f81-a0d9-699893d31743 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:18:56.162586 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1324/2616] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:18:56 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:18:56.166377 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-306e9539-1cae-4067-9b06-e897bc2d7e6a admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:18:56.176153 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-306e9539-1cae-4067-9b06-e897bc2d7e6a admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:18:56.180723 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d1e3f00-05cd-4543-9778-203f9ec0f393 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:18:56.181349 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d1e3f00-05cd-4543-9778-203f9ec0f393 admin admin] REQUEST_METHOD: `PUT` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:56.181625 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d1e3f00-05cd-4543-9778-203f9ec0f393 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:56.181895 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d1e3f00-05cd-4543-9778-203f9ec0f393 admin admin] PATH_INFO: `/v3/projects/f3fdad6f801941b7923342ffe7cb0818/users/44cd028d17584e5797f55e614a5f4db0/roles/6f03d86980c34c36b44a8d6764ff4fda` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:56.196207 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d1e3f00-05cd-4543-9778-203f9ec0f393 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f3fdad6f801941b7923342ffe7cb0818, user_id=44cd028d17584e5797f55e614a5f4db0, role_id=6f03d86980c34c36b44a8d6764ff4fda)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:18:56.198846 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d1e3f00-05cd-4543-9778-203f9ec0f393 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:18:56.206240 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=85,keystone:DELETE=11,keystone:INSERT=4 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:18:56.212407 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1293/2617] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:18:56 2022] PUT /identity/v3/projects/f3fdad6f801941b7923342ffe7cb0818/users/44cd028d17584e5797f55e614a5f4db0/roles/6f03d86980c34c36b44a8d6764ff4fda => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:18:56.222704 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfafa2a7-4a6f-4ff3-81ac-18270aa63eb7 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:56.222970 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfafa2a7-4a6f-4ff3-81ac-18270aa63eb7 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:56.223239 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfafa2a7-4a6f-4ff3-81ac-18270aa63eb7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:56.250474 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-dfafa2a7-4a6f-4ff3-81ac-18270aa63eb7 None None] MFA Rules not processed for user `44cd028d17584e5797f55e614a5f4db0`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:18:56.251696 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-dfafa2a7-4a6f-4ff3-81ac-18270aa63eb7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:18:56.293638 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:18:56.316550 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1325/2618] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:18:56 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:18:56.328061 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d1e3f00-05cd-4543-9778-203f9ec0f393 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:18:56.329717 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-5d1e3f00-05cd-4543-9778-203f9ec0f393 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:18:56.386981 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d1e3f00-05cd-4543-9778-203f9ec0f393 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:18:56.390518 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1de55fa8-078b-4c5e-a88d-eeda3585ec92 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a002ac0bf240a891ac7304178d3f0e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:18:56.391134 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de55fa8-078b-4c5e-a88d-eeda3585ec92 service glance] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:18:56.391409 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de55fa8-078b-4c5e-a88d-eeda3585ec92 service glance] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:18:56.391680 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de55fa8-078b-4c5e-a88d-eeda3585ec92 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:18:56.399719 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1de55fa8-078b-4c5e-a88d-eeda3585ec92 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:18:56.401140 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1de55fa8-078b-4c5e-a88d-eeda3585ec92 service glance] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:18:56.409719 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1294/2619] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:18:56 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 82 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:19:06.313711 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=8 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:19:06.405873 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=27 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:19:09.725600 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfafa2a7-4a6f-4ff3-81ac-18270aa63eb7 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:09.727613 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-dfafa2a7-4a6f-4ff3-81ac-18270aa63eb7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:19:09.790348 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfafa2a7-4a6f-4ff3-81ac-18270aa63eb7 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:09.794012 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-782a9155-c185-44f3-927a-c4828e1907a1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:09.795129 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-782a9155-c185-44f3-927a-c4828e1907a1 service neutron] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:09.795401 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-782a9155-c185-44f3-927a-c4828e1907a1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:09.796307 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-782a9155-c185-44f3-927a-c4828e1907a1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:09.798008 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-782a9155-c185-44f3-927a-c4828e1907a1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:19:09.864990 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-782a9155-c185-44f3-927a-c4828e1907a1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:09.866598 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-782a9155-c185-44f3-927a-c4828e1907a1 service neutron] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:09.879413 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1326/2620] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:19:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 154 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 05:19:13.164691 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1de55fa8-078b-4c5e-a88d-eeda3585ec92 service glance] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:13.166277 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-1de55fa8-078b-4c5e-a88d-eeda3585ec92 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:19:13.196421 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1de55fa8-078b-4c5e-a88d-eeda3585ec92 service glance] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:13.201844 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-850b7754-9681-4ebb-ab69-5246cc5e21bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:13.202495 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-850b7754-9681-4ebb-ab69-5246cc5e21bd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:13.202767 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-850b7754-9681-4ebb-ab69-5246cc5e21bd admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:13.203069 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-850b7754-9681-4ebb-ab69-5246cc5e21bd admin admin] PATH_INFO: `/v3/users/2fe451987a8c49c5b5fc0ef1c9c30ab5` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:13.204173 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-850b7754-9681-4ebb-ab69-5246cc5e21bd admin admin] RBAC: Authorizing `identity:delete_user(user_id=2fe451987a8c49c5b5fc0ef1c9c30ab5)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:13.206067 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-850b7754-9681-4ebb-ab69-5246cc5e21bd admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:13.255522 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-850b7754-9681-4ebb-ab69-5246cc5e21bd admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2fe451987a8c49c5b5fc0ef1c9c30ab5'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:13.256176 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-850b7754-9681-4ebb-ab69-5246cc5e21bd admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2fe451987a8c49c5b5fc0ef1c9c30ab5'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:13.256854 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-850b7754-9681-4ebb-ab69-5246cc5e21bd admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2fe451987a8c49c5b5fc0ef1c9c30ab5'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:13.269326 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-850b7754-9681-4ebb-ab69-5246cc5e21bd admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2fe451987a8c49c5b5fc0ef1c9c30ab5'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:13.276330 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-850b7754-9681-4ebb-ab69-5246cc5e21bd admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2fe451987a8c49c5b5fc0ef1c9c30ab5'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:13.289514 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1295/2621] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:19:13 2022] DELETE /identity/v3/users/2fe451987a8c49c5b5fc0ef1c9c30ab5 => generated 0 bytes in 125 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:19:13.467086 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-782a9155-c185-44f3-927a-c4828e1907a1 service neutron] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:13.477443 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-782a9155-c185-44f3-927a-c4828e1907a1 service neutron] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:13.536460 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fec6435-8e20-4a16-8070-9fa7c5920bf4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:13.537087 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fec6435-8e20-4a16-8070-9fa7c5920bf4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:13.537366 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fec6435-8e20-4a16-8070-9fa7c5920bf4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:13.537665 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fec6435-8e20-4a16-8070-9fa7c5920bf4 admin admin] PATH_INFO: `/v3/projects/7ea4da0dbf3a44eabbff29f884beb157` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:13.538768 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fec6435-8e20-4a16-8070-9fa7c5920bf4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7ea4da0dbf3a44eabbff29f884beb157)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:13.540701 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fec6435-8e20-4a16-8070-9fa7c5920bf4 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:13.586105 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-8fec6435-8e20-4a16-8070-9fa7c5920bf4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7ea4da0dbf3a44eabbff29f884beb157', 'request_id': 'req-8fec6435-8e20-4a16-8070-9fa7c5920bf4'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:13.588964 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=95,keystone:DELETE=4,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:19:13.594897 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-8fec6435-8e20-4a16-8070-9fa7c5920bf4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7ea4da0dbf3a44eabbff29f884beb157', 'request_id': 'req-8fec6435-8e20-4a16-8070-9fa7c5920bf4'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:13.602785 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-8fec6435-8e20-4a16-8070-9fa7c5920bf4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7ea4da0dbf3a44eabbff29f884beb157', 'request_id': 'req-8fec6435-8e20-4a16-8070-9fa7c5920bf4'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:13.613064 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-8fec6435-8e20-4a16-8070-9fa7c5920bf4 admin admin] The token cache is being invalidate because project 7ea4da0dbf3a44eabbff29f884beb157 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81625) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:19:13.613354 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-8fec6435-8e20-4a16-8070-9fa7c5920bf4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:13.613979 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-8fec6435-8e20-4a16-8070-9fa7c5920bf4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:13.615499 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1327/2622] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:19:13 2022] DELETE /identity/v3/projects/7ea4da0dbf3a44eabbff29f884beb157 => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:19:13.620510 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b748b67-f2e0-43e0-a7d7-0ab6d6e42f78 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:13.620801 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b748b67-f2e0-43e0-a7d7-0ab6d6e42f78 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:13.621081 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b748b67-f2e0-43e0-a7d7-0ab6d6e42f78 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:13.643983 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-4b748b67-f2e0-43e0-a7d7-0ab6d6e42f78 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:19:13.645193 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-4b748b67-f2e0-43e0-a7d7-0ab6d6e42f78 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:19:13.703379 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1296/2623] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:19:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:19:13.711447 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d279eecb-60df-41f8-bdb4-6e96e2dda251 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:13.711750 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d279eecb-60df-41f8-bdb4-6e96e2dda251 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:13.712023 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d279eecb-60df-41f8-bdb4-6e96e2dda251 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:13.731727 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-d279eecb-60df-41f8-bdb4-6e96e2dda251 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:19:13.732920 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-d279eecb-60df-41f8-bdb4-6e96e2dda251 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:19:13.740722 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1328/2624] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:19:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:19:13.744846 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b748b67-f2e0-43e0-a7d7-0ab6d6e42f78 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:13.755166 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b748b67-f2e0-43e0-a7d7-0ab6d6e42f78 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:13.759668 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c78d84c-fe98-4dc6-95a3-5f38cc1f4694 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:13.760253 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c78d84c-fe98-4dc6-95a3-5f38cc1f4694 admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:13.760531 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c78d84c-fe98-4dc6-95a3-5f38cc1f4694 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:13.760802 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c78d84c-fe98-4dc6-95a3-5f38cc1f4694 admin admin] PATH_INFO: `/v3/domains` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:13.761446 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c78d84c-fe98-4dc6-95a3-5f38cc1f4694 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:19:13.761746 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c78d84c-fe98-4dc6-95a3-5f38cc1f4694 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:13.763978 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c78d84c-fe98-4dc6-95a3-5f38cc1f4694 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:13.775265 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1297/2625] 10.222.0.77 () {58 vars in 1153 bytes} [Tue Aug 30 05:19:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:19:13.779462 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d279eecb-60df-41f8-bdb4-6e96e2dda251 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:13.788524 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d279eecb-60df-41f8-bdb4-6e96e2dda251 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:13.794038 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c3044f6-0242-4e2f-a84d-6002cdae14d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:13.794633 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c3044f6-0242-4e2f-a84d-6002cdae14d6 admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:13.794949 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c3044f6-0242-4e2f-a84d-6002cdae14d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:13.795221 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c3044f6-0242-4e2f-a84d-6002cdae14d6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:13.795843 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c3044f6-0242-4e2f-a84d-6002cdae14d6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:13.797750 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c3044f6-0242-4e2f-a84d-6002cdae14d6 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:13.824913 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1329/2626] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:19:13 2022] POST /identity/v3/projects => generated 373 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:19:13.829533 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c78d84c-fe98-4dc6-95a3-5f38cc1f4694 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:13.839358 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c78d84c-fe98-4dc6-95a3-5f38cc1f4694 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:13.894798 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13d12810-772f-45ec-a9cf-282c476d0ef3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:13.895331 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13d12810-772f-45ec-a9cf-282c476d0ef3 admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:13.895950 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13d12810-772f-45ec-a9cf-282c476d0ef3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:13.896230 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13d12810-772f-45ec-a9cf-282c476d0ef3 admin admin] PATH_INFO: `/v3/users` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:13.896854 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13d12810-772f-45ec-a9cf-282c476d0ef3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:13.898770 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13d12810-772f-45ec-a9cf-282c476d0ef3 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:13.906683 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:19:13.921682 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1298/2627] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:19:13 2022] POST /identity/v3/users => generated 287 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:19:13.925775 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c3044f6-0242-4e2f-a84d-6002cdae14d6 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:13.936606 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c3044f6-0242-4e2f-a84d-6002cdae14d6 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:13.941313 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cadb2a73-2893-4789-b6c3-84ad70e8b899 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:13.941900 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cadb2a73-2893-4789-b6c3-84ad70e8b899 admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:13.942198 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cadb2a73-2893-4789-b6c3-84ad70e8b899 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:13.942470 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cadb2a73-2893-4789-b6c3-84ad70e8b899 admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:13.943131 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cadb2a73-2893-4789-b6c3-84ad70e8b899 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:13.944754 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cadb2a73-2893-4789-b6c3-84ad70e8b899 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:13.954617 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1330/2628] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:19:13 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:19:13.958137 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13d12810-772f-45ec-a9cf-282c476d0ef3 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:13.968211 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13d12810-772f-45ec-a9cf-282c476d0ef3 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:13.972742 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-690a5785-3b27-475f-ba19-f0fc2f833e92 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:13.973320 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-690a5785-3b27-475f-ba19-f0fc2f833e92 admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:13.973598 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-690a5785-3b27-475f-ba19-f0fc2f833e92 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:13.973870 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-690a5785-3b27-475f-ba19-f0fc2f833e92 admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:13.974539 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-690a5785-3b27-475f-ba19-f0fc2f833e92 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:13.976142 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-690a5785-3b27-475f-ba19-f0fc2f833e92 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:13.988054 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1299/2629] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:19:13 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:19:13.991860 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cadb2a73-2893-4789-b6c3-84ad70e8b899 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:14.002104 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cadb2a73-2893-4789-b6c3-84ad70e8b899 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:14.006706 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51170f59-a950-449c-8df9-5c8134f43af2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:14.007377 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51170f59-a950-449c-8df9-5c8134f43af2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:14.007653 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51170f59-a950-449c-8df9-5c8134f43af2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:14.007924 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51170f59-a950-449c-8df9-5c8134f43af2 admin admin] PATH_INFO: `/v3/projects/90e437bbf3774a7daafd6fa6eaa1e00e/users/6a00d2571c1c46ee92251172223709a9/roles/6f03d86980c34c36b44a8d6764ff4fda` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:14.022468 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51170f59-a950-449c-8df9-5c8134f43af2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=90e437bbf3774a7daafd6fa6eaa1e00e, user_id=6a00d2571c1c46ee92251172223709a9, role_id=6f03d86980c34c36b44a8d6764ff4fda)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:14.025155 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51170f59-a950-449c-8df9-5c8134f43af2 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:14.037868 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1331/2630] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:19:13 2022] PUT /identity/v3/projects/90e437bbf3774a7daafd6fa6eaa1e00e/users/6a00d2571c1c46ee92251172223709a9/roles/6f03d86980c34c36b44a8d6764ff4fda => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:19:14.049151 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b54a6b4-0387-43ac-89a1-b0a15352868e None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:14.049435 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b54a6b4-0387-43ac-89a1-b0a15352868e None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:14.049708 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b54a6b4-0387-43ac-89a1-b0a15352868e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:14.077366 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-6b54a6b4-0387-43ac-89a1-b0a15352868e None None] MFA Rules not processed for user `6a00d2571c1c46ee92251172223709a9`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:19:14.078575 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-6b54a6b4-0387-43ac-89a1-b0a15352868e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:19:14.144385 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1300/2631] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:19:14 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:19:14.154766 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51170f59-a950-449c-8df9-5c8134f43af2 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:14.156498 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-51170f59-a950-449c-8df9-5c8134f43af2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:19:14.228754 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51170f59-a950-449c-8df9-5c8134f43af2 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:14.234744 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c1d022b-ff97-4d6e-8f3c-0766f6a9a9f3 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '886eb5e00bbf4ec292eb962b665b37d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:14.235363 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c1d022b-ff97-4d6e-8f3c-0766f6a9a9f3 service nova] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:14.235644 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c1d022b-ff97-4d6e-8f3c-0766f6a9a9f3 service nova] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:14.235914 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c1d022b-ff97-4d6e-8f3c-0766f6a9a9f3 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:14.243434 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c1d022b-ff97-4d6e-8f3c-0766f6a9a9f3 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:14.245101 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c1d022b-ff97-4d6e-8f3c-0766f6a9a9f3 service nova] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:14.254495 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1332/2632] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:19:14 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:19:18.783265 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da4869d1-bb88-4421-b39d-8d1217729a00 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:18.783755 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da4869d1-bb88-4421-b39d-8d1217729a00 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:18.783852 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da4869d1-bb88-4421-b39d-8d1217729a00 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:18.803252 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-da4869d1-bb88-4421-b39d-8d1217729a00 None None] MFA Rules not processed for user `d079e0264f0e40db9fc93b8f217a7a74`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:19:18.804467 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-da4869d1-bb88-4421-b39d-8d1217729a00 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:19:18.859601 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1301/2633] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:19:18 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 77 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:19:19.059156 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c1d022b-ff97-4d6e-8f3c-0766f6a9a9f3 service nova] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:19.121998 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c1d022b-ff97-4d6e-8f3c-0766f6a9a9f3 service nova] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:19.127082 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6dc974d-faea-461e-9fc6-06c1c2709e0b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:19.127709 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6dc974d-faea-461e-9fc6-06c1c2709e0b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:19.127986 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6dc974d-faea-461e-9fc6-06c1c2709e0b admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:19.128252 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6dc974d-faea-461e-9fc6-06c1c2709e0b admin admin] PATH_INFO: `/v3/users/6a00d2571c1c46ee92251172223709a9` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:19.129356 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6dc974d-faea-461e-9fc6-06c1c2709e0b admin admin] RBAC: Authorizing `identity:delete_user(user_id=6a00d2571c1c46ee92251172223709a9)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:19.131306 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6dc974d-faea-461e-9fc6-06c1c2709e0b admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:19.133845 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:DELETE=3,keystone:SELECT=95,keystone:INSERT=2 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:19:19.182780 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-a6dc974d-faea-461e-9fc6-06c1c2709e0b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6a00d2571c1c46ee92251172223709a9'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:19.190888 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-a6dc974d-faea-461e-9fc6-06c1c2709e0b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6a00d2571c1c46ee92251172223709a9'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:19.203481 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-a6dc974d-faea-461e-9fc6-06c1c2709e0b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6a00d2571c1c46ee92251172223709a9'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:19.210491 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-a6dc974d-faea-461e-9fc6-06c1c2709e0b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6a00d2571c1c46ee92251172223709a9'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:19.211272 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-a6dc974d-faea-461e-9fc6-06c1c2709e0b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6a00d2571c1c46ee92251172223709a9'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:19.218316 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1333/2634] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:19:19 2022] DELETE /identity/v3/users/6a00d2571c1c46ee92251172223709a9 => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:19:19.222829 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8dcadd0-4b28-464d-9ca9-2cce6319c63d None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:19.223103 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8dcadd0-4b28-464d-9ca9-2cce6319c63d None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:19.223377 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8dcadd0-4b28-464d-9ca9-2cce6319c63d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:19.241886 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-a8dcadd0-4b28-464d-9ca9-2cce6319c63d None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:19:19.243233 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-a8dcadd0-4b28-464d-9ca9-2cce6319c63d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:19:19.301943 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1302/2635] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:19:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:19:19.312591 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6dc974d-faea-461e-9fc6-06c1c2709e0b admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:19.314018 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-a6dc974d-faea-461e-9fc6-06c1c2709e0b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:19:19.378159 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6dc974d-faea-461e-9fc6-06c1c2709e0b admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:19.381920 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f68c292c-1e38-4be7-a936-7d244b6fd9f1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:19.382514 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f68c292c-1e38-4be7-a936-7d244b6fd9f1 service neutron] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:19.382800 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f68c292c-1e38-4be7-a936-7d244b6fd9f1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:19.383098 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f68c292c-1e38-4be7-a936-7d244b6fd9f1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:19.393763 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f68c292c-1e38-4be7-a936-7d244b6fd9f1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:19.395251 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f68c292c-1e38-4be7-a936-7d244b6fd9f1 service neutron] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:19.404968 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1334/2636] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:19:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:19:19.765237 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8dcadd0-4b28-464d-9ca9-2cce6319c63d None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:19.767122 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-a8dcadd0-4b28-464d-9ca9-2cce6319c63d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:19:19.785647 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8dcadd0-4b28-464d-9ca9-2cce6319c63d None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:19.798207 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd65053f-9e7d-4de3-9cd9-89333dd82be5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:19.799022 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd65053f-9e7d-4de3-9cd9-89333dd82be5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:19.799485 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd65053f-9e7d-4de3-9cd9-89333dd82be5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:19.799767 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd65053f-9e7d-4de3-9cd9-89333dd82be5 admin admin] PATH_INFO: `/v3/projects/90e437bbf3774a7daafd6fa6eaa1e00e` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:19.801347 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd65053f-9e7d-4de3-9cd9-89333dd82be5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=90e437bbf3774a7daafd6fa6eaa1e00e)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:19.804723 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd65053f-9e7d-4de3-9cd9-89333dd82be5 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:19.816016 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:INSERT=3,keystone:SELECT=97 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:19:19.865498 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-cd65053f-9e7d-4de3-9cd9-89333dd82be5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '90e437bbf3774a7daafd6fa6eaa1e00e', 'request_id': 'req-cd65053f-9e7d-4de3-9cd9-89333dd82be5'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:19.875837 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-cd65053f-9e7d-4de3-9cd9-89333dd82be5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '90e437bbf3774a7daafd6fa6eaa1e00e', 'request_id': 'req-cd65053f-9e7d-4de3-9cd9-89333dd82be5'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:19.882310 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-cd65053f-9e7d-4de3-9cd9-89333dd82be5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '90e437bbf3774a7daafd6fa6eaa1e00e', 'request_id': 'req-cd65053f-9e7d-4de3-9cd9-89333dd82be5'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:19.895232 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-cd65053f-9e7d-4de3-9cd9-89333dd82be5 admin admin] The token cache is being invalidate because project 90e437bbf3774a7daafd6fa6eaa1e00e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81624) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:19:19.895618 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-cd65053f-9e7d-4de3-9cd9-89333dd82be5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:19.896345 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-cd65053f-9e7d-4de3-9cd9-89333dd82be5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:19.898587 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1303/2637] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:19:19 2022] DELETE /identity/v3/projects/90e437bbf3774a7daafd6fa6eaa1e00e => generated 0 bytes in 134 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:19:19.903925 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18aaea0c-5c9b-4ae9-a095-430eb76bbb3c None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:19.904297 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18aaea0c-5c9b-4ae9-a095-430eb76bbb3c None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:19.904703 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18aaea0c-5c9b-4ae9-a095-430eb76bbb3c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:19.929684 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-18aaea0c-5c9b-4ae9-a095-430eb76bbb3c None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:19:19.931471 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-18aaea0c-5c9b-4ae9-a095-430eb76bbb3c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:19:20.003778 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1335/2638] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:19:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:19:20.012551 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d6f0e4-a4ff-4888-92a1-ff68b4381398 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:20.012882 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d6f0e4-a4ff-4888-92a1-ff68b4381398 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:20.013134 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d6f0e4-a4ff-4888-92a1-ff68b4381398 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:20.035962 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-91d6f0e4-a4ff-4888-92a1-ff68b4381398 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:19:20.037245 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-91d6f0e4-a4ff-4888-92a1-ff68b4381398 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:19:20.053368 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1304/2639] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:19:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:19:20.058088 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18aaea0c-5c9b-4ae9-a095-430eb76bbb3c None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:20.077288 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18aaea0c-5c9b-4ae9-a095-430eb76bbb3c None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:20.086412 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4ecd02d-7ad7-4a54-9a9c-62cb22b6e7e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:20.087696 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4ecd02d-7ad7-4a54-9a9c-62cb22b6e7e2 admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:20.088402 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4ecd02d-7ad7-4a54-9a9c-62cb22b6e7e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:20.089343 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4ecd02d-7ad7-4a54-9a9c-62cb22b6e7e2 admin admin] PATH_INFO: `/v3/domains` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:20.090993 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4ecd02d-7ad7-4a54-9a9c-62cb22b6e7e2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:19:20.091496 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4ecd02d-7ad7-4a54-9a9c-62cb22b6e7e2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:20.094786 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4ecd02d-7ad7-4a54-9a9c-62cb22b6e7e2 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:20.127137 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1336/2640] 10.222.0.77 () {58 vars in 1153 bytes} [Tue Aug 30 05:19:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 70 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:19:20.128261 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91d6f0e4-a4ff-4888-92a1-ff68b4381398 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:20.139196 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91d6f0e4-a4ff-4888-92a1-ff68b4381398 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:20.144831 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3329eb65-3eca-4248-bb3b-3fb9de4d098e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:20.145548 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3329eb65-3eca-4248-bb3b-3fb9de4d098e admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:20.145890 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3329eb65-3eca-4248-bb3b-3fb9de4d098e admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:20.146240 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3329eb65-3eca-4248-bb3b-3fb9de4d098e admin admin] PATH_INFO: `/v3/projects` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:20.147171 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3329eb65-3eca-4248-bb3b-3fb9de4d098e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:20.149232 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3329eb65-3eca-4248-bb3b-3fb9de4d098e admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:20.179859 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1305/2641] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:19:20 2022] POST /identity/v3/projects => generated 371 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 05:19:20.184175 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4ecd02d-7ad7-4a54-9a9c-62cb22b6e7e2 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:20.194531 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4ecd02d-7ad7-4a54-9a9c-62cb22b6e7e2 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:20.202773 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:19:20.256812 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5755a062-c1cd-4002-9dd2-c2c6f6047322 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:20.257414 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5755a062-c1cd-4002-9dd2-c2c6f6047322 admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:20.257755 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5755a062-c1cd-4002-9dd2-c2c6f6047322 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:20.258027 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5755a062-c1cd-4002-9dd2-c2c6f6047322 admin admin] PATH_INFO: `/v3/users` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:20.258700 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5755a062-c1cd-4002-9dd2-c2c6f6047322 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:20.260902 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5755a062-c1cd-4002-9dd2-c2c6f6047322 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:20.286093 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1337/2642] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:19:20 2022] POST /identity/v3/users => generated 286 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 05:19:20.287360 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3329eb65-3eca-4248-bb3b-3fb9de4d098e admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:20.297543 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3329eb65-3eca-4248-bb3b-3fb9de4d098e admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:20.302052 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbe6349d-f994-478b-a230-5a5a8045685a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:20.302666 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbe6349d-f994-478b-a230-5a5a8045685a admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:20.302980 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbe6349d-f994-478b-a230-5a5a8045685a admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:20.303246 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbe6349d-f994-478b-a230-5a5a8045685a admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:20.303870 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbe6349d-f994-478b-a230-5a5a8045685a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:20.305459 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbe6349d-f994-478b-a230-5a5a8045685a admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:20.315302 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1306/2643] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:19:20 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:19:20.319097 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5755a062-c1cd-4002-9dd2-c2c6f6047322 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:20.329329 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5755a062-c1cd-4002-9dd2-c2c6f6047322 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:20.333898 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66d3e00d-02bb-445b-9e84-a7c708a649f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:20.334481 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d3e00d-02bb-445b-9e84-a7c708a649f2 admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:20.334761 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d3e00d-02bb-445b-9e84-a7c708a649f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:20.335064 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d3e00d-02bb-445b-9e84-a7c708a649f2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:20.335697 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66d3e00d-02bb-445b-9e84-a7c708a649f2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:20.337269 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66d3e00d-02bb-445b-9e84-a7c708a649f2 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:20.348423 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1338/2644] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:19:20 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:19:20.352417 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbe6349d-f994-478b-a230-5a5a8045685a admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:20.362277 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbe6349d-f994-478b-a230-5a5a8045685a admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:20.367059 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f228445-683c-49f2-a229-3d8295177f2f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:20.367695 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f228445-683c-49f2-a229-3d8295177f2f admin admin] REQUEST_METHOD: `PUT` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:20.367973 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f228445-683c-49f2-a229-3d8295177f2f admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:20.368241 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f228445-683c-49f2-a229-3d8295177f2f admin admin] PATH_INFO: `/v3/projects/0e57e0cf02db48d387c7091122d1e1af/users/d44e8106431b44a48b231c6b83edfef6/roles/6f03d86980c34c36b44a8d6764ff4fda` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:20.382625 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f228445-683c-49f2-a229-3d8295177f2f admin admin] RBAC: Authorizing `identity:create_grant(project_id=0e57e0cf02db48d387c7091122d1e1af, user_id=d44e8106431b44a48b231c6b83edfef6, role_id=6f03d86980c34c36b44a8d6764ff4fda)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:20.385287 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f228445-683c-49f2-a229-3d8295177f2f admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:20.398288 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1307/2645] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:19:20 2022] PUT /identity/v3/projects/0e57e0cf02db48d387c7091122d1e1af/users/d44e8106431b44a48b231c6b83edfef6/roles/6f03d86980c34c36b44a8d6764ff4fda => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:19:20.409196 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dd1af22-936c-48e5-8bec-b08bf998ecdc None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:20.409488 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dd1af22-936c-48e5-8bec-b08bf998ecdc None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:20.409764 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dd1af22-936c-48e5-8bec-b08bf998ecdc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:20.436629 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-5dd1af22-936c-48e5-8bec-b08bf998ecdc None None] MFA Rules not processed for user `d44e8106431b44a48b231c6b83edfef6`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:19:20.437825 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-5dd1af22-936c-48e5-8bec-b08bf998ecdc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:19:20.502767 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1339/2646] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:19:20 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:19:20.515297 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f228445-683c-49f2-a229-3d8295177f2f admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:20.517118 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-9f228445-683c-49f2-a229-3d8295177f2f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:19:20.591754 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f228445-683c-49f2-a229-3d8295177f2f admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:20.599290 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73672263-3632-4f0d-bba8-f3d5a6f2f959 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '886eb5e00bbf4ec292eb962b665b37d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:20.599886 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73672263-3632-4f0d-bba8-f3d5a6f2f959 service nova] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:20.600162 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73672263-3632-4f0d-bba8-f3d5a6f2f959 service nova] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:20.600430 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73672263-3632-4f0d-bba8-f3d5a6f2f959 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:20.608635 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73672263-3632-4f0d-bba8-f3d5a6f2f959 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:20.610449 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73672263-3632-4f0d-bba8-f3d5a6f2f959 service nova] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:20.620092 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1308/2647] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:19:20 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:19:20.829094 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dd1af22-936c-48e5-8bec-b08bf998ecdc None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:20.889890 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dd1af22-936c-48e5-8bec-b08bf998ecdc None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:20.894342 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ce5803a-87f9-4efe-ad9f-6b8c7713ff05 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:20.894996 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ce5803a-87f9-4efe-ad9f-6b8c7713ff05 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:20.895273 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ce5803a-87f9-4efe-ad9f-6b8c7713ff05 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:20.895542 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ce5803a-87f9-4efe-ad9f-6b8c7713ff05 admin admin] PATH_INFO: `/v3/users/d44e8106431b44a48b231c6b83edfef6` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:20.896647 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ce5803a-87f9-4efe-ad9f-6b8c7713ff05 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d44e8106431b44a48b231c6b83edfef6)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:20.898549 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ce5803a-87f9-4efe-ad9f-6b8c7713ff05 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:20.919041 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=94,keystone:INSERT=3,keystone:DELETE=3 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:19:20.945321 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-9ce5803a-87f9-4efe-ad9f-6b8c7713ff05 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd44e8106431b44a48b231c6b83edfef6'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:20.952913 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-9ce5803a-87f9-4efe-ad9f-6b8c7713ff05 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd44e8106431b44a48b231c6b83edfef6'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:20.964837 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-9ce5803a-87f9-4efe-ad9f-6b8c7713ff05 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd44e8106431b44a48b231c6b83edfef6'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:20.971965 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-9ce5803a-87f9-4efe-ad9f-6b8c7713ff05 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd44e8106431b44a48b231c6b83edfef6'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:20.972601 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-9ce5803a-87f9-4efe-ad9f-6b8c7713ff05 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd44e8106431b44a48b231c6b83edfef6'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:20.978392 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1340/2648] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:19:20 2022] DELETE /identity/v3/users/d44e8106431b44a48b231c6b83edfef6 => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:19:20.983086 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-555b769e-9492-49e7-8572-5d87494c8744 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:20.983550 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-555b769e-9492-49e7-8572-5d87494c8744 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:20.983663 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-555b769e-9492-49e7-8572-5d87494c8744 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:21.003497 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-555b769e-9492-49e7-8572-5d87494c8744 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:19:21.004720 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-555b769e-9492-49e7-8572-5d87494c8744 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:19:21.019422 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:19:21.068780 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1309/2649] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:19:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:19:21.078483 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ce5803a-87f9-4efe-ad9f-6b8c7713ff05 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:21.079969 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-9ce5803a-87f9-4efe-ad9f-6b8c7713ff05 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:19:21.137673 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ce5803a-87f9-4efe-ad9f-6b8c7713ff05 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:21.141298 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c1b37bb-708a-443e-a3d0-d76d042c8da9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:21.141898 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c1b37bb-708a-443e-a3d0-d76d042c8da9 service neutron] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:21.142179 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c1b37bb-708a-443e-a3d0-d76d042c8da9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:21.142447 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c1b37bb-708a-443e-a3d0-d76d042c8da9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:21.151854 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c1b37bb-708a-443e-a3d0-d76d042c8da9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:21.153306 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c1b37bb-708a-443e-a3d0-d76d042c8da9 service neutron] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:21.162888 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1341/2650] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:19:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:19:21.423290 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-555b769e-9492-49e7-8572-5d87494c8744 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:21.424707 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-555b769e-9492-49e7-8572-5d87494c8744 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:19:21.441326 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-555b769e-9492-49e7-8572-5d87494c8744 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:21.445997 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1005c89-8ee5-428a-8b22-84aa5fcdeddd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:21.446666 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1005c89-8ee5-428a-8b22-84aa5fcdeddd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:21.447054 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1005c89-8ee5-428a-8b22-84aa5fcdeddd admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:21.447200 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1005c89-8ee5-428a-8b22-84aa5fcdeddd admin admin] PATH_INFO: `/v3/projects/0e57e0cf02db48d387c7091122d1e1af` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:21.448305 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1005c89-8ee5-428a-8b22-84aa5fcdeddd admin admin] RBAC: Authorizing `identity:delete_project(project_id=0e57e0cf02db48d387c7091122d1e1af)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:21.450231 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1005c89-8ee5-428a-8b22-84aa5fcdeddd admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:21.497361 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-d1005c89-8ee5-428a-8b22-84aa5fcdeddd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0e57e0cf02db48d387c7091122d1e1af', 'request_id': 'req-d1005c89-8ee5-428a-8b22-84aa5fcdeddd'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:21.504464 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-d1005c89-8ee5-428a-8b22-84aa5fcdeddd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0e57e0cf02db48d387c7091122d1e1af', 'request_id': 'req-d1005c89-8ee5-428a-8b22-84aa5fcdeddd'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:21.511110 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-d1005c89-8ee5-428a-8b22-84aa5fcdeddd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0e57e0cf02db48d387c7091122d1e1af', 'request_id': 'req-d1005c89-8ee5-428a-8b22-84aa5fcdeddd'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:21.521524 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-d1005c89-8ee5-428a-8b22-84aa5fcdeddd admin admin] The token cache is being invalidate because project 0e57e0cf02db48d387c7091122d1e1af was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81624) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:19:21.521811 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-d1005c89-8ee5-428a-8b22-84aa5fcdeddd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:21.522461 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-d1005c89-8ee5-428a-8b22-84aa5fcdeddd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:21.523973 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1310/2651] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:19:21 2022] DELETE /identity/v3/projects/0e57e0cf02db48d387c7091122d1e1af => generated 0 bytes in 101 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:19:21.528987 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ec16051-9de8-419c-8621-d6353db00e75 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:21.529278 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ec16051-9de8-419c-8621-d6353db00e75 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:21.529544 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ec16051-9de8-419c-8621-d6353db00e75 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:21.549552 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-0ec16051-9de8-419c-8621-d6353db00e75 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:19:21.550734 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-0ec16051-9de8-419c-8621-d6353db00e75 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:19:21.612161 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1342/2652] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:19:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:19:21.620303 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca22bd3-ddc4-46ce-9e2d-bed9d5bc3653 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:21.620579 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca22bd3-ddc4-46ce-9e2d-bed9d5bc3653 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:21.620859 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca22bd3-ddc4-46ce-9e2d-bed9d5bc3653 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:21.640237 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-5ca22bd3-ddc4-46ce-9e2d-bed9d5bc3653 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:19:21.641434 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-5ca22bd3-ddc4-46ce-9e2d-bed9d5bc3653 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:19:21.649344 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1311/2653] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:19:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:19:21.653433 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ec16051-9de8-419c-8621-d6353db00e75 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:21.663789 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ec16051-9de8-419c-8621-d6353db00e75 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:21.668331 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-478151d2-3b2c-418a-8602-72ad0984e76f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:21.668911 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478151d2-3b2c-418a-8602-72ad0984e76f admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:21.669197 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478151d2-3b2c-418a-8602-72ad0984e76f admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:21.669453 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478151d2-3b2c-418a-8602-72ad0984e76f admin admin] PATH_INFO: `/v3/domains` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:21.670084 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-478151d2-3b2c-418a-8602-72ad0984e76f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:19:21.670402 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-478151d2-3b2c-418a-8602-72ad0984e76f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:21.672517 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-478151d2-3b2c-418a-8602-72ad0984e76f admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:21.684223 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1343/2654] 10.222.0.77 () {58 vars in 1153 bytes} [Tue Aug 30 05:19:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:19:21.688400 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ca22bd3-ddc4-46ce-9e2d-bed9d5bc3653 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:21.698062 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ca22bd3-ddc4-46ce-9e2d-bed9d5bc3653 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:21.702799 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b72d17b0-4a25-44e3-8aee-a43d4d73339b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:21.703437 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b72d17b0-4a25-44e3-8aee-a43d4d73339b admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:21.703710 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b72d17b0-4a25-44e3-8aee-a43d4d73339b admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:21.703997 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b72d17b0-4a25-44e3-8aee-a43d4d73339b admin admin] PATH_INFO: `/v3/projects` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:21.704612 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b72d17b0-4a25-44e3-8aee-a43d4d73339b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:21.706554 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b72d17b0-4a25-44e3-8aee-a43d4d73339b admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:21.737991 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1312/2655] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:19:21 2022] POST /identity/v3/projects => generated 371 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:19:21.739217 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-478151d2-3b2c-418a-8602-72ad0984e76f admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:21.749394 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-478151d2-3b2c-418a-8602-72ad0984e76f admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:21.776781 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:DELETE=7,keystone:SELECT=92,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:19:21.809031 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a7d83bc-007b-48a6-a9fe-8206402e071b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:21.809620 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a7d83bc-007b-48a6-a9fe-8206402e071b admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:21.809893 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a7d83bc-007b-48a6-a9fe-8206402e071b admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:21.810158 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a7d83bc-007b-48a6-a9fe-8206402e071b admin admin] PATH_INFO: `/v3/users` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:21.810778 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a7d83bc-007b-48a6-a9fe-8206402e071b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:21.812729 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a7d83bc-007b-48a6-a9fe-8206402e071b admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:21.833211 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1344/2656] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:19:21 2022] POST /identity/v3/users => generated 286 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 05:19:21.837462 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b72d17b0-4a25-44e3-8aee-a43d4d73339b admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:21.848700 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b72d17b0-4a25-44e3-8aee-a43d4d73339b admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:21.853252 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e48fabd-83bc-4077-99eb-28816810994e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:21.853833 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e48fabd-83bc-4077-99eb-28816810994e admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:21.854106 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e48fabd-83bc-4077-99eb-28816810994e admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:21.854369 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e48fabd-83bc-4077-99eb-28816810994e admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:21.855020 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e48fabd-83bc-4077-99eb-28816810994e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:21.856622 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e48fabd-83bc-4077-99eb-28816810994e admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:21.866446 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1313/2657] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:19:21 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:19:21.869995 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a7d83bc-007b-48a6-a9fe-8206402e071b admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:21.879592 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a7d83bc-007b-48a6-a9fe-8206402e071b admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:21.884180 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c8e4b99-9390-491a-99b5-2421ecd4376a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:21.884763 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c8e4b99-9390-491a-99b5-2421ecd4376a admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:21.885030 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c8e4b99-9390-491a-99b5-2421ecd4376a admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:21.885298 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c8e4b99-9390-491a-99b5-2421ecd4376a admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:21.885920 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c8e4b99-9390-491a-99b5-2421ecd4376a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:21.887524 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c8e4b99-9390-491a-99b5-2421ecd4376a admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:21.899287 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1345/2658] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:19:21 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:19:21.903105 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e48fabd-83bc-4077-99eb-28816810994e admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:21.913296 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e48fabd-83bc-4077-99eb-28816810994e admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:21.917884 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1855362d-f3bd-4ef8-bafd-265a86c42333 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:21.918499 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1855362d-f3bd-4ef8-bafd-265a86c42333 admin admin] REQUEST_METHOD: `PUT` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:21.918798 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1855362d-f3bd-4ef8-bafd-265a86c42333 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:21.919114 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1855362d-f3bd-4ef8-bafd-265a86c42333 admin admin] PATH_INFO: `/v3/projects/d7bb3c7164ab4739af6e770aaa081c14/users/f0ed75663f754fbc80a579d3fdc78863/roles/6f03d86980c34c36b44a8d6764ff4fda` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:21.933620 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1855362d-f3bd-4ef8-bafd-265a86c42333 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d7bb3c7164ab4739af6e770aaa081c14, user_id=f0ed75663f754fbc80a579d3fdc78863, role_id=6f03d86980c34c36b44a8d6764ff4fda)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:21.936427 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1855362d-f3bd-4ef8-bafd-265a86c42333 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:21.948827 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1314/2659] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:19:21 2022] PUT /identity/v3/projects/d7bb3c7164ab4739af6e770aaa081c14/users/f0ed75663f754fbc80a579d3fdc78863/roles/6f03d86980c34c36b44a8d6764ff4fda => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:19:21.960760 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c20f5b1-1a22-48cf-b1e8-7792139c5662 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:21.961049 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c20f5b1-1a22-48cf-b1e8-7792139c5662 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:21.961322 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c20f5b1-1a22-48cf-b1e8-7792139c5662 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:21.988795 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-6c20f5b1-1a22-48cf-b1e8-7792139c5662 None None] MFA Rules not processed for user `f0ed75663f754fbc80a579d3fdc78863`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:19:21.989995 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-6c20f5b1-1a22-48cf-b1e8-7792139c5662 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:19:22.052942 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1346/2660] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:19:21 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:19:22.057567 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1855362d-f3bd-4ef8-bafd-265a86c42333 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:22.098622 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:19:22.120863 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1855362d-f3bd-4ef8-bafd-265a86c42333 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:22.126066 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adf07a8f-b094-4816-8cf3-ee7aaea73d3b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:22.126664 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adf07a8f-b094-4816-8cf3-ee7aaea73d3b admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:22.126976 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adf07a8f-b094-4816-8cf3-ee7aaea73d3b admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:22.127251 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adf07a8f-b094-4816-8cf3-ee7aaea73d3b admin admin] PATH_INFO: `/v3/projects` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:22.127881 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adf07a8f-b094-4816-8cf3-ee7aaea73d3b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:22.129819 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adf07a8f-b094-4816-8cf3-ee7aaea73d3b admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:22.155673 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1315/2661] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:19:22 2022] POST /identity/v3/projects => generated 373 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:19:22.159810 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c20f5b1-1a22-48cf-b1e8-7792139c5662 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:22.171275 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c20f5b1-1a22-48cf-b1e8-7792139c5662 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:22.229376 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6323032-1e55-4de4-babb-3d3edaa9de90 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:22.229978 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6323032-1e55-4de4-babb-3d3edaa9de90 admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:22.230253 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6323032-1e55-4de4-babb-3d3edaa9de90 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:22.230520 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6323032-1e55-4de4-babb-3d3edaa9de90 admin admin] PATH_INFO: `/v3/users` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:22.231182 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6323032-1e55-4de4-babb-3d3edaa9de90 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:22.233070 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6323032-1e55-4de4-babb-3d3edaa9de90 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:22.253529 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1347/2662] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:19:22 2022] POST /identity/v3/users => generated 293 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:19:22.257522 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adf07a8f-b094-4816-8cf3-ee7aaea73d3b admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:22.268475 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adf07a8f-b094-4816-8cf3-ee7aaea73d3b admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:22.273111 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6886063-4c5f-4f3e-b1e4-707cb8d43111 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:22.273700 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6886063-4c5f-4f3e-b1e4-707cb8d43111 admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:22.273981 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6886063-4c5f-4f3e-b1e4-707cb8d43111 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:22.274254 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6886063-4c5f-4f3e-b1e4-707cb8d43111 admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:22.274923 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6886063-4c5f-4f3e-b1e4-707cb8d43111 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:22.276508 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6886063-4c5f-4f3e-b1e4-707cb8d43111 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:22.286954 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1316/2663] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:19:22 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:19:22.290831 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6323032-1e55-4de4-babb-3d3edaa9de90 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:22.300491 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6323032-1e55-4de4-babb-3d3edaa9de90 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:22.305085 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a1a796d-9354-4d9f-8d0c-80d6d6134dca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:22.305712 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a1a796d-9354-4d9f-8d0c-80d6d6134dca admin admin] REQUEST_METHOD: `PUT` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:22.305988 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a1a796d-9354-4d9f-8d0c-80d6d6134dca admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:22.306257 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a1a796d-9354-4d9f-8d0c-80d6d6134dca admin admin] PATH_INFO: `/v3/projects/df25ef1e0e32437ebd44d7fdbfdbb032/users/3fbd46bc2f4f435c8438983e206511f1/roles/9d9d5d2c270146488249b7ed943280db` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:22.321421 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a1a796d-9354-4d9f-8d0c-80d6d6134dca admin admin] RBAC: Authorizing `identity:create_grant(project_id=df25ef1e0e32437ebd44d7fdbfdbb032, user_id=3fbd46bc2f4f435c8438983e206511f1, role_id=9d9d5d2c270146488249b7ed943280db)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:22.324151 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a1a796d-9354-4d9f-8d0c-80d6d6134dca admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:22.337201 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1348/2664] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:19:22 2022] PUT /identity/v3/projects/df25ef1e0e32437ebd44d7fdbfdbb032/users/3fbd46bc2f4f435c8438983e206511f1/roles/9d9d5d2c270146488249b7ed943280db => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:19:22.341260 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6886063-4c5f-4f3e-b1e4-707cb8d43111 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:22.351215 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6886063-4c5f-4f3e-b1e4-707cb8d43111 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:22.406454 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8396681e-56a7-4894-b6eb-5b453da4eaf4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:22.407010 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8396681e-56a7-4894-b6eb-5b453da4eaf4 admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:22.407321 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8396681e-56a7-4894-b6eb-5b453da4eaf4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:22.407592 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8396681e-56a7-4894-b6eb-5b453da4eaf4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:22.408221 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8396681e-56a7-4894-b6eb-5b453da4eaf4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:22.409802 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8396681e-56a7-4894-b6eb-5b453da4eaf4 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:22.419347 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1317/2665] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:19:22 2022] GET /identity/v3/roles => generated 1420 bytes in 79 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:19:22.423529 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a1a796d-9354-4d9f-8d0c-80d6d6134dca admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:22.428897 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:19:22.435653 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a1a796d-9354-4d9f-8d0c-80d6d6134dca admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:22.441217 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-019f908d-5aea-4fa2-9653-3507999a6fc5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:22.441861 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-019f908d-5aea-4fa2-9653-3507999a6fc5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:22.442143 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-019f908d-5aea-4fa2-9653-3507999a6fc5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:22.442418 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-019f908d-5aea-4fa2-9653-3507999a6fc5 admin admin] PATH_INFO: `/v3/projects/df25ef1e0e32437ebd44d7fdbfdbb032/users/3fbd46bc2f4f435c8438983e206511f1/roles/9d9d5d2c270146488249b7ed943280db` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:22.444391 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-019f908d-5aea-4fa2-9653-3507999a6fc5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=df25ef1e0e32437ebd44d7fdbfdbb032, user_id=3fbd46bc2f4f435c8438983e206511f1, role_id=9d9d5d2c270146488249b7ed943280db)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:22.447060 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-019f908d-5aea-4fa2-9653-3507999a6fc5 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:22.458132 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1349/2666] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:19:22 2022] PUT /identity/v3/projects/df25ef1e0e32437ebd44d7fdbfdbb032/users/3fbd46bc2f4f435c8438983e206511f1/roles/9d9d5d2c270146488249b7ed943280db => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:19:22.468536 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bbb48f5-3231-4a5f-b9a5-6266b0286510 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:22.468869 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bbb48f5-3231-4a5f-b9a5-6266b0286510 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:22.469162 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bbb48f5-3231-4a5f-b9a5-6266b0286510 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:22.496488 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-1bbb48f5-3231-4a5f-b9a5-6266b0286510 None None] MFA Rules not processed for user `3fbd46bc2f4f435c8438983e206511f1`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:19:22.497697 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-1bbb48f5-3231-4a5f-b9a5-6266b0286510 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:19:22.566660 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1318/2667] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:19:22 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:19:22.578065 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-019f908d-5aea-4fa2-9653-3507999a6fc5 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:22.579775 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-019f908d-5aea-4fa2-9653-3507999a6fc5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:19:22.637327 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-019f908d-5aea-4fa2-9653-3507999a6fc5 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:22.640929 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a28d00da-5b24-427f-9238-455762d91744 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '019a79b2affe478b8a30d1c538d0415c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:22.641556 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a28d00da-5b24-427f-9238-455762d91744 service cinder] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:22.641823 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a28d00da-5b24-427f-9238-455762d91744 service cinder] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:22.642094 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a28d00da-5b24-427f-9238-455762d91744 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:22.651035 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a28d00da-5b24-427f-9238-455762d91744 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:22.652441 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a28d00da-5b24-427f-9238-455762d91744 service cinder] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:22.661783 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1350/2668] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:19:22 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:19:23.044885 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bbb48f5-3231-4a5f-b9a5-6266b0286510 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:23.080803 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:19:23.114350 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bbb48f5-3231-4a5f-b9a5-6266b0286510 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:23.119241 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae10e2f3-5bdd-4f91-b7a7-a0b3395b5062 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:23.119861 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae10e2f3-5bdd-4f91-b7a7-a0b3395b5062 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:23.120139 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae10e2f3-5bdd-4f91-b7a7-a0b3395b5062 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:23.120412 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae10e2f3-5bdd-4f91-b7a7-a0b3395b5062 admin admin] PATH_INFO: `/v3/users/f0ed75663f754fbc80a579d3fdc78863` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:23.121615 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae10e2f3-5bdd-4f91-b7a7-a0b3395b5062 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f0ed75663f754fbc80a579d3fdc78863)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:23.123567 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae10e2f3-5bdd-4f91-b7a7-a0b3395b5062 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:23.178788 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-ae10e2f3-5bdd-4f91-b7a7-a0b3395b5062 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f0ed75663f754fbc80a579d3fdc78863'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:23.179513 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-ae10e2f3-5bdd-4f91-b7a7-a0b3395b5062 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f0ed75663f754fbc80a579d3fdc78863'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:23.180164 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-ae10e2f3-5bdd-4f91-b7a7-a0b3395b5062 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f0ed75663f754fbc80a579d3fdc78863'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:23.194015 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-ae10e2f3-5bdd-4f91-b7a7-a0b3395b5062 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f0ed75663f754fbc80a579d3fdc78863'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:23.202245 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-ae10e2f3-5bdd-4f91-b7a7-a0b3395b5062 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f0ed75663f754fbc80a579d3fdc78863'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:23.215108 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1319/2669] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:19:23 2022] DELETE /identity/v3/users/f0ed75663f754fbc80a579d3fdc78863 => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:19:23.220440 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc1784db-7644-4d20-b8ee-663f5ed8e651 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:23.220837 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc1784db-7644-4d20-b8ee-663f5ed8e651 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:23.221032 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc1784db-7644-4d20-b8ee-663f5ed8e651 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:23.242213 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-dc1784db-7644-4d20-b8ee-663f5ed8e651 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:19:23.244009 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-dc1784db-7644-4d20-b8ee-663f5ed8e651 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:19:23.313716 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1351/2670] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:19:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:19:23.323151 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae10e2f3-5bdd-4f91-b7a7-a0b3395b5062 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:23.324556 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-ae10e2f3-5bdd-4f91-b7a7-a0b3395b5062 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:19:23.382991 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae10e2f3-5bdd-4f91-b7a7-a0b3395b5062 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:23.386639 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-722dff14-8e1e-4b2c-9545-8eb55e212332 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:23.387272 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-722dff14-8e1e-4b2c-9545-8eb55e212332 service neutron] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:23.387552 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-722dff14-8e1e-4b2c-9545-8eb55e212332 service neutron] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:23.387823 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-722dff14-8e1e-4b2c-9545-8eb55e212332 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:23.398576 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-722dff14-8e1e-4b2c-9545-8eb55e212332 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:23.400088 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-722dff14-8e1e-4b2c-9545-8eb55e212332 service neutron] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:23.410581 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1320/2671] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:19:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:19:23.982765 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc1784db-7644-4d20-b8ee-663f5ed8e651 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:23.984251 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-dc1784db-7644-4d20-b8ee-663f5ed8e651 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:19:23.996374 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc1784db-7644-4d20-b8ee-663f5ed8e651 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:24.001809 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-008ca684-9e6a-492f-adb3-2a021fe59d3b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:24.002516 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-008ca684-9e6a-492f-adb3-2a021fe59d3b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:24.002693 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-008ca684-9e6a-492f-adb3-2a021fe59d3b admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:24.003024 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-008ca684-9e6a-492f-adb3-2a021fe59d3b admin admin] PATH_INFO: `/v3/projects/d7bb3c7164ab4739af6e770aaa081c14` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:24.004106 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-008ca684-9e6a-492f-adb3-2a021fe59d3b admin admin] RBAC: Authorizing `identity:delete_project(project_id=d7bb3c7164ab4739af6e770aaa081c14)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:24.005990 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-008ca684-9e6a-492f-adb3-2a021fe59d3b admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:24.062498 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-008ca684-9e6a-492f-adb3-2a021fe59d3b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd7bb3c7164ab4739af6e770aaa081c14', 'request_id': 'req-008ca684-9e6a-492f-adb3-2a021fe59d3b'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:24.072524 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-008ca684-9e6a-492f-adb3-2a021fe59d3b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd7bb3c7164ab4739af6e770aaa081c14', 'request_id': 'req-008ca684-9e6a-492f-adb3-2a021fe59d3b'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:24.080597 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-008ca684-9e6a-492f-adb3-2a021fe59d3b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd7bb3c7164ab4739af6e770aaa081c14', 'request_id': 'req-008ca684-9e6a-492f-adb3-2a021fe59d3b'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:24.092916 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-008ca684-9e6a-492f-adb3-2a021fe59d3b admin admin] The token cache is being invalidate because project d7bb3c7164ab4739af6e770aaa081c14 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81625) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:19:24.093224 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-008ca684-9e6a-492f-adb3-2a021fe59d3b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:24.093870 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-008ca684-9e6a-492f-adb3-2a021fe59d3b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:24.098682 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1352/2672] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:19:23 2022] DELETE /identity/v3/projects/d7bb3c7164ab4739af6e770aaa081c14 => generated 0 bytes in 113 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:19:24.099688 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-722dff14-8e1e-4b2c-9545-8eb55e212332 service neutron] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:24.112263 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-722dff14-8e1e-4b2c-9545-8eb55e212332 service neutron] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:24.171776 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:19:24.179924 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f6a3249-1ec7-49c1-9212-d188ee3e7b2f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:24.180698 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f6a3249-1ec7-49c1-9212-d188ee3e7b2f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:24.180994 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f6a3249-1ec7-49c1-9212-d188ee3e7b2f admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:24.181701 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f6a3249-1ec7-49c1-9212-d188ee3e7b2f admin admin] PATH_INFO: `/v3/users/3fbd46bc2f4f435c8438983e206511f1` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:24.183560 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f6a3249-1ec7-49c1-9212-d188ee3e7b2f admin admin] RBAC: Authorizing `identity:delete_user(user_id=3fbd46bc2f4f435c8438983e206511f1)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:24.185850 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f6a3249-1ec7-49c1-9212-d188ee3e7b2f admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:24.241259 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-6f6a3249-1ec7-49c1-9212-d188ee3e7b2f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3fbd46bc2f4f435c8438983e206511f1'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:24.241974 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-6f6a3249-1ec7-49c1-9212-d188ee3e7b2f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3fbd46bc2f4f435c8438983e206511f1'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:24.242778 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-6f6a3249-1ec7-49c1-9212-d188ee3e7b2f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3fbd46bc2f4f435c8438983e206511f1'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:24.259075 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-6f6a3249-1ec7-49c1-9212-d188ee3e7b2f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3fbd46bc2f4f435c8438983e206511f1'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:24.266260 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-6f6a3249-1ec7-49c1-9212-d188ee3e7b2f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3fbd46bc2f4f435c8438983e206511f1'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:24.279638 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1321/2673] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:19:24 2022] DELETE /identity/v3/users/3fbd46bc2f4f435c8438983e206511f1 => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:19:24.770841 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-008ca684-9e6a-492f-adb3-2a021fe59d3b admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:24.772371 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-008ca684-9e6a-492f-adb3-2a021fe59d3b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:19:24.786614 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-008ca684-9e6a-492f-adb3-2a021fe59d3b admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:24.799425 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:19:24.852856 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f18efca7-c0eb-4fe7-9e6c-05fd4eeef1b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:24.853518 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f18efca7-c0eb-4fe7-9e6c-05fd4eeef1b5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:24.853798 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f18efca7-c0eb-4fe7-9e6c-05fd4eeef1b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:24.854070 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f18efca7-c0eb-4fe7-9e6c-05fd4eeef1b5 admin admin] PATH_INFO: `/v3/projects/df25ef1e0e32437ebd44d7fdbfdbb032` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:24.855457 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f18efca7-c0eb-4fe7-9e6c-05fd4eeef1b5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=df25ef1e0e32437ebd44d7fdbfdbb032)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:24.857473 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f18efca7-c0eb-4fe7-9e6c-05fd4eeef1b5 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:24.908687 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-f18efca7-c0eb-4fe7-9e6c-05fd4eeef1b5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'df25ef1e0e32437ebd44d7fdbfdbb032', 'request_id': 'req-f18efca7-c0eb-4fe7-9e6c-05fd4eeef1b5'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:24.916802 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-f18efca7-c0eb-4fe7-9e6c-05fd4eeef1b5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'df25ef1e0e32437ebd44d7fdbfdbb032', 'request_id': 'req-f18efca7-c0eb-4fe7-9e6c-05fd4eeef1b5'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:24.923984 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-f18efca7-c0eb-4fe7-9e6c-05fd4eeef1b5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'df25ef1e0e32437ebd44d7fdbfdbb032', 'request_id': 'req-f18efca7-c0eb-4fe7-9e6c-05fd4eeef1b5'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:24.935201 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-f18efca7-c0eb-4fe7-9e6c-05fd4eeef1b5 admin admin] The token cache is being invalidate because project df25ef1e0e32437ebd44d7fdbfdbb032 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81625) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:19:24.935492 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-f18efca7-c0eb-4fe7-9e6c-05fd4eeef1b5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:24.936327 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-f18efca7-c0eb-4fe7-9e6c-05fd4eeef1b5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:24.937836 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1353/2674] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:19:24 2022] DELETE /identity/v3/projects/df25ef1e0e32437ebd44d7fdbfdbb032 => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:19:24.943920 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18b8da43-cceb-49a6-9407-81ce0a64f522 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:24.944311 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18b8da43-cceb-49a6-9407-81ce0a64f522 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:24.944657 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18b8da43-cceb-49a6-9407-81ce0a64f522 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:24.965777 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-18b8da43-cceb-49a6-9407-81ce0a64f522 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:19:24.967164 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-18b8da43-cceb-49a6-9407-81ce0a64f522 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:19:25.034268 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1322/2675] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:19:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:19:25.043718 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37feca31-a12e-40f1-836b-56d90f398387 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:25.044040 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37feca31-a12e-40f1-836b-56d90f398387 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:25.044278 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37feca31-a12e-40f1-836b-56d90f398387 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:25.065699 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-37feca31-a12e-40f1-836b-56d90f398387 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:19:25.066956 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-37feca31-a12e-40f1-836b-56d90f398387 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:19:25.075237 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1354/2676] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:19:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:19:25.079883 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18b8da43-cceb-49a6-9407-81ce0a64f522 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:25.091997 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18b8da43-cceb-49a6-9407-81ce0a64f522 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:25.104861 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03e0ca8d-6b9c-4e93-a2b4-b12ce6d35eb2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:25.105515 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03e0ca8d-6b9c-4e93-a2b4-b12ce6d35eb2 admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:25.105938 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03e0ca8d-6b9c-4e93-a2b4-b12ce6d35eb2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:25.106332 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03e0ca8d-6b9c-4e93-a2b4-b12ce6d35eb2 admin admin] PATH_INFO: `/v3/domains` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:25.107159 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03e0ca8d-6b9c-4e93-a2b4-b12ce6d35eb2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:19:25.117576 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03e0ca8d-6b9c-4e93-a2b4-b12ce6d35eb2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:25.120310 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03e0ca8d-6b9c-4e93-a2b4-b12ce6d35eb2 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:25.147326 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1323/2677] 10.222.0.77 () {58 vars in 1153 bytes} [Tue Aug 30 05:19:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 68 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:19:25.148591 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37feca31-a12e-40f1-836b-56d90f398387 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:25.160515 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37feca31-a12e-40f1-836b-56d90f398387 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:25.165751 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bf2c480-a93a-48dc-a4a6-42c00648ba83 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:25.166407 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bf2c480-a93a-48dc-a4a6-42c00648ba83 admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:25.166688 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bf2c480-a93a-48dc-a4a6-42c00648ba83 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:25.166987 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bf2c480-a93a-48dc-a4a6-42c00648ba83 admin admin] PATH_INFO: `/v3/projects` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:25.168343 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bf2c480-a93a-48dc-a4a6-42c00648ba83 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:25.170721 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bf2c480-a93a-48dc-a4a6-42c00648ba83 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:25.214743 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1355/2678] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:19:25 2022] POST /identity/v3/projects => generated 409 bytes in 68 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:19:25.219190 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03e0ca8d-6b9c-4e93-a2b4-b12ce6d35eb2 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:25.229190 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03e0ca8d-6b9c-4e93-a2b4-b12ce6d35eb2 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:25.289918 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccb97e0c-9cc0-4ea5-b6cc-e3c1bfb08575 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:25.290552 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccb97e0c-9cc0-4ea5-b6cc-e3c1bfb08575 admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:25.290850 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccb97e0c-9cc0-4ea5-b6cc-e3c1bfb08575 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:25.291124 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccb97e0c-9cc0-4ea5-b6cc-e3c1bfb08575 admin admin] PATH_INFO: `/v3/users` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:25.291753 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccb97e0c-9cc0-4ea5-b6cc-e3c1bfb08575 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:25.293657 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccb97e0c-9cc0-4ea5-b6cc-e3c1bfb08575 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:25.302657 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:19:25.319355 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1324/2679] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:19:25 2022] POST /identity/v3/users => generated 305 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 05:19:25.320891 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bf2c480-a93a-48dc-a4a6-42c00648ba83 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:25.332160 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bf2c480-a93a-48dc-a4a6-42c00648ba83 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:25.336746 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b008a268-861a-49ff-a8de-f8a0046ac9e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:25.337344 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b008a268-861a-49ff-a8de-f8a0046ac9e1 admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:25.337639 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b008a268-861a-49ff-a8de-f8a0046ac9e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:25.337893 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b008a268-861a-49ff-a8de-f8a0046ac9e1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:25.338526 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b008a268-861a-49ff-a8de-f8a0046ac9e1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:25.340438 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b008a268-861a-49ff-a8de-f8a0046ac9e1 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:25.477754 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1356/2680] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:19:25 2022] GET /identity/v3/roles => generated 1420 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:19:25.481730 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccb97e0c-9cc0-4ea5-b6cc-e3c1bfb08575 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:25.491761 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccb97e0c-9cc0-4ea5-b6cc-e3c1bfb08575 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:25.496427 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c54adb91-86d8-451b-be31-6039816a0442 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:25.497021 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c54adb91-86d8-451b-be31-6039816a0442 admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:25.497295 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c54adb91-86d8-451b-be31-6039816a0442 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:25.497560 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c54adb91-86d8-451b-be31-6039816a0442 admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:25.498187 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c54adb91-86d8-451b-be31-6039816a0442 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:25.500357 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c54adb91-86d8-451b-be31-6039816a0442 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:25.511100 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1325/2681] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:19:25 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:19:25.515760 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b008a268-861a-49ff-a8de-f8a0046ac9e1 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:25.525793 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b008a268-861a-49ff-a8de-f8a0046ac9e1 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:25.530293 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e9e1295-583c-41dd-85f8-e410cac2927e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:25.530954 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e9e1295-583c-41dd-85f8-e410cac2927e admin admin] REQUEST_METHOD: `PUT` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:25.531225 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e9e1295-583c-41dd-85f8-e410cac2927e admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:25.531487 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e9e1295-583c-41dd-85f8-e410cac2927e admin admin] PATH_INFO: `/v3/projects/7b41626669874461afac002a46ee26d7/users/ccb0e0ccc7ec4495a6d5676419103273/roles/6f03d86980c34c36b44a8d6764ff4fda` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:25.546253 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e9e1295-583c-41dd-85f8-e410cac2927e admin admin] RBAC: Authorizing `identity:create_grant(project_id=7b41626669874461afac002a46ee26d7, user_id=ccb0e0ccc7ec4495a6d5676419103273, role_id=6f03d86980c34c36b44a8d6764ff4fda)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:25.548973 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e9e1295-583c-41dd-85f8-e410cac2927e admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:25.568883 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1357/2682] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:19:25 2022] PUT /identity/v3/projects/7b41626669874461afac002a46ee26d7/users/ccb0e0ccc7ec4495a6d5676419103273/roles/6f03d86980c34c36b44a8d6764ff4fda => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 05:19:25.579655 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5451f45-1a4a-44c9-8ee1-f5765d00ef3f None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:25.580131 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5451f45-1a4a-44c9-8ee1-f5765d00ef3f None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:25.580385 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5451f45-1a4a-44c9-8ee1-f5765d00ef3f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:25.610720 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-b5451f45-1a4a-44c9-8ee1-f5765d00ef3f None None] MFA Rules not processed for user `ccb0e0ccc7ec4495a6d5676419103273`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:19:25.612379 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-b5451f45-1a4a-44c9-8ee1-f5765d00ef3f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:19:25.683263 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1326/2683] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:19:25 2022] POST /identity/v3/auth/tokens => generated 3286 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:19:25.688398 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e9e1295-583c-41dd-85f8-e410cac2927e admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:25.732020 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:19:25.752505 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e9e1295-583c-41dd-85f8-e410cac2927e admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:25.757164 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c37656ee-ca79-4abd-a8be-0d67c8c5a33d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:25.757750 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c37656ee-ca79-4abd-a8be-0d67c8c5a33d admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:25.758038 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c37656ee-ca79-4abd-a8be-0d67c8c5a33d admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:25.758303 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c37656ee-ca79-4abd-a8be-0d67c8c5a33d admin admin] PATH_INFO: `/v3/projects` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:25.758971 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c37656ee-ca79-4abd-a8be-0d67c8c5a33d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:25.760876 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c37656ee-ca79-4abd-a8be-0d67c8c5a33d admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:25.787221 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1358/2684] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:19:25 2022] POST /identity/v3/projects => generated 409 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:19:25.791644 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5451f45-1a4a-44c9-8ee1-f5765d00ef3f None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:25.803146 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5451f45-1a4a-44c9-8ee1-f5765d00ef3f None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:25.863384 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e98a9cc2-1177-40fd-95bb-edac311a47ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:25.864005 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e98a9cc2-1177-40fd-95bb-edac311a47ce admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:25.864310 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e98a9cc2-1177-40fd-95bb-edac311a47ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:25.864619 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e98a9cc2-1177-40fd-95bb-edac311a47ce admin admin] PATH_INFO: `/v3/users` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:25.865285 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e98a9cc2-1177-40fd-95bb-edac311a47ce admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:25.867375 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e98a9cc2-1177-40fd-95bb-edac311a47ce admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:25.888584 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1327/2685] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:19:25 2022] POST /identity/v3/users => generated 311 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:19:25.892689 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c37656ee-ca79-4abd-a8be-0d67c8c5a33d admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:25.902893 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c37656ee-ca79-4abd-a8be-0d67c8c5a33d admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:25.907994 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-582beee0-30ea-461d-9cac-36ece5c02b97 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:25.908586 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-582beee0-30ea-461d-9cac-36ece5c02b97 admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:25.908866 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-582beee0-30ea-461d-9cac-36ece5c02b97 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:25.909141 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-582beee0-30ea-461d-9cac-36ece5c02b97 admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:25.909764 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-582beee0-30ea-461d-9cac-36ece5c02b97 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:25.911777 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-582beee0-30ea-461d-9cac-36ece5c02b97 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:25.922409 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1359/2686] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:19:25 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:19:25.926324 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e98a9cc2-1177-40fd-95bb-edac311a47ce admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:25.936278 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e98a9cc2-1177-40fd-95bb-edac311a47ce admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:25.941126 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7507aa74-8e73-4c52-9b91-0442dbbeb2dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:25.941808 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7507aa74-8e73-4c52-9b91-0442dbbeb2dc admin admin] REQUEST_METHOD: `PUT` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:25.942092 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7507aa74-8e73-4c52-9b91-0442dbbeb2dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:25.942369 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7507aa74-8e73-4c52-9b91-0442dbbeb2dc admin admin] PATH_INFO: `/v3/projects/8f1968c6ac6643f1910ba73ae236f23c/users/972195d72ba0472a84c98c4f897b36c1/roles/9d9d5d2c270146488249b7ed943280db` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:25.956895 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7507aa74-8e73-4c52-9b91-0442dbbeb2dc admin admin] RBAC: Authorizing `identity:create_grant(project_id=8f1968c6ac6643f1910ba73ae236f23c, user_id=972195d72ba0472a84c98c4f897b36c1, role_id=9d9d5d2c270146488249b7ed943280db)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:25.959608 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7507aa74-8e73-4c52-9b91-0442dbbeb2dc admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:25.973133 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1328/2687] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:19:25 2022] PUT /identity/v3/projects/8f1968c6ac6643f1910ba73ae236f23c/users/972195d72ba0472a84c98c4f897b36c1/roles/9d9d5d2c270146488249b7ed943280db => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:19:25.977047 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-582beee0-30ea-461d-9cac-36ece5c02b97 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:25.987499 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-582beee0-30ea-461d-9cac-36ece5c02b97 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:26.043174 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-250778ba-0fdd-4de6-8327-b8da3e39980f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:26.043765 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-250778ba-0fdd-4de6-8327-b8da3e39980f admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:26.044043 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-250778ba-0fdd-4de6-8327-b8da3e39980f admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:26.044311 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-250778ba-0fdd-4de6-8327-b8da3e39980f admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:26.044939 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-250778ba-0fdd-4de6-8327-b8da3e39980f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:26.047104 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-250778ba-0fdd-4de6-8327-b8da3e39980f admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:26.056747 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1360/2688] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:19:25 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 05:19:26.060805 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7507aa74-8e73-4c52-9b91-0442dbbeb2dc admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:26.071050 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7507aa74-8e73-4c52-9b91-0442dbbeb2dc admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:26.075649 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e50e032f-6b5d-4691-9bca-9a01ca0afba3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:26.076287 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e50e032f-6b5d-4691-9bca-9a01ca0afba3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:26.076601 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e50e032f-6b5d-4691-9bca-9a01ca0afba3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:26.076920 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e50e032f-6b5d-4691-9bca-9a01ca0afba3 admin admin] PATH_INFO: `/v3/projects/8f1968c6ac6643f1910ba73ae236f23c/users/972195d72ba0472a84c98c4f897b36c1/roles/9d9d5d2c270146488249b7ed943280db` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:26.078835 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e50e032f-6b5d-4691-9bca-9a01ca0afba3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8f1968c6ac6643f1910ba73ae236f23c, user_id=972195d72ba0472a84c98c4f897b36c1, role_id=9d9d5d2c270146488249b7ed943280db)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:26.081462 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e50e032f-6b5d-4691-9bca-9a01ca0afba3 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:26.091754 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1329/2689] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:19:26 2022] PUT /identity/v3/projects/8f1968c6ac6643f1910ba73ae236f23c/users/972195d72ba0472a84c98c4f897b36c1/roles/9d9d5d2c270146488249b7ed943280db => generated 0 bytes in 31 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:19:26.102296 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6e615e3-4843-4c9e-aac9-c587ee7fbafc None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:26.102579 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6e615e3-4843-4c9e-aac9-c587ee7fbafc None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:26.102906 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6e615e3-4843-4c9e-aac9-c587ee7fbafc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:26.130123 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-f6e615e3-4843-4c9e-aac9-c587ee7fbafc None None] MFA Rules not processed for user `972195d72ba0472a84c98c4f897b36c1`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:19:26.131361 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-f6e615e3-4843-4c9e-aac9-c587ee7fbafc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:19:26.197442 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1361/2690] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:19:26 2022] POST /identity/v3/auth/tokens => generated 3353 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:19:26.207757 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e50e032f-6b5d-4691-9bca-9a01ca0afba3 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:26.209615 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-e50e032f-6b5d-4691-9bca-9a01ca0afba3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:19:26.239091 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:INSERT=7,keystone:SELECT=93 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:19:26.271545 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e50e032f-6b5d-4691-9bca-9a01ca0afba3 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:26.275192 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a67af069-53b8-4324-9b79-26a12d9b5f41 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '019a79b2affe478b8a30d1c538d0415c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:26.275793 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67af069-53b8-4324-9b79-26a12d9b5f41 service cinder] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:26.276071 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67af069-53b8-4324-9b79-26a12d9b5f41 service cinder] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:26.276342 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67af069-53b8-4324-9b79-26a12d9b5f41 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:26.286053 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a67af069-53b8-4324-9b79-26a12d9b5f41 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:26.287870 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a67af069-53b8-4324-9b79-26a12d9b5f41 service cinder] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:26.298168 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1330/2691] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:19:26 2022] GET /identity/v3/auth/tokens => generated 3353 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:19:26.445419 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6e615e3-4843-4c9e-aac9-c587ee7fbafc None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:26.455363 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6e615e3-4843-4c9e-aac9-c587ee7fbafc None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:26.458910 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b5b2bea-2a38-4ca1-ad0c-6a1db464084f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '019a79b2affe478b8a30d1c538d0415c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:26.459501 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5b2bea-2a38-4ca1-ad0c-6a1db464084f service cinder] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:26.459774 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5b2bea-2a38-4ca1-ad0c-6a1db464084f service cinder] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:26.460036 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5b2bea-2a38-4ca1-ad0c-6a1db464084f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:26.494295 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:19:26.518924 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b5b2bea-2a38-4ca1-ad0c-6a1db464084f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:26.520371 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b5b2bea-2a38-4ca1-ad0c-6a1db464084f service cinder] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:26.529256 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1362/2692] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:19:26 2022] GET /identity/v3/auth/tokens => generated 3286 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:19:28.998218 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a67af069-53b8-4324-9b79-26a12d9b5f41 service cinder] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:28.999647 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-a67af069-53b8-4324-9b79-26a12d9b5f41 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:19:29.063151 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a67af069-53b8-4324-9b79-26a12d9b5f41 service cinder] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:29.067631 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bab4a45-c953-4f77-aa0f-f7bd95049316 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:29.068238 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bab4a45-c953-4f77-aa0f-f7bd95049316 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:29.068513 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bab4a45-c953-4f77-aa0f-f7bd95049316 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:29.068778 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bab4a45-c953-4f77-aa0f-f7bd95049316 admin admin] PATH_INFO: `/v3/users/44cd028d17584e5797f55e614a5f4db0` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:29.069862 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bab4a45-c953-4f77-aa0f-f7bd95049316 admin admin] RBAC: Authorizing `identity:delete_user(user_id=44cd028d17584e5797f55e614a5f4db0)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:29.071794 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bab4a45-c953-4f77-aa0f-f7bd95049316 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:29.119969 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-9bab4a45-c953-4f77-aa0f-f7bd95049316 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '44cd028d17584e5797f55e614a5f4db0'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:29.120641 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-9bab4a45-c953-4f77-aa0f-f7bd95049316 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '44cd028d17584e5797f55e614a5f4db0'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:29.121240 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-9bab4a45-c953-4f77-aa0f-f7bd95049316 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '44cd028d17584e5797f55e614a5f4db0'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:29.132994 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-9bab4a45-c953-4f77-aa0f-f7bd95049316 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '44cd028d17584e5797f55e614a5f4db0'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:29.139879 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-9bab4a45-c953-4f77-aa0f-f7bd95049316 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '44cd028d17584e5797f55e614a5f4db0'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:29.152002 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1331/2693] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:19:28 2022] DELETE /identity/v3/users/44cd028d17584e5797f55e614a5f4db0 => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:19:29.156866 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26fe373e-5a41-4c8c-8f17-38db25e9cc98 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:29.157345 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26fe373e-5a41-4c8c-8f17-38db25e9cc98 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:29.157475 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26fe373e-5a41-4c8c-8f17-38db25e9cc98 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:29.176372 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-26fe373e-5a41-4c8c-8f17-38db25e9cc98 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:19:29.177556 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-26fe373e-5a41-4c8c-8f17-38db25e9cc98 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:19:29.235416 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1363/2694] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:19:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 05:19:29.244878 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bab4a45-c953-4f77-aa0f-f7bd95049316 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:29.246299 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-9bab4a45-c953-4f77-aa0f-f7bd95049316 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:19:29.292554 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:19:29.304274 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bab4a45-c953-4f77-aa0f-f7bd95049316 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:29.308029 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-008eb8cd-32c6-4b3f-87ee-924dce1409bf service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:29.308620 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-008eb8cd-32c6-4b3f-87ee-924dce1409bf service neutron] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:29.308922 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-008eb8cd-32c6-4b3f-87ee-924dce1409bf service neutron] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:29.309188 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-008eb8cd-32c6-4b3f-87ee-924dce1409bf service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:29.318040 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-008eb8cd-32c6-4b3f-87ee-924dce1409bf service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:29.319487 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-008eb8cd-32c6-4b3f-87ee-924dce1409bf service neutron] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:29.329709 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1332/2695] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:19:29 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:19:29.593741 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26fe373e-5a41-4c8c-8f17-38db25e9cc98 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:29.595312 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-26fe373e-5a41-4c8c-8f17-38db25e9cc98 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:19:29.607442 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26fe373e-5a41-4c8c-8f17-38db25e9cc98 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:29.612067 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdff00ba-2596-47bd-a64a-39dd98aa0788 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:29.612686 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdff00ba-2596-47bd-a64a-39dd98aa0788 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:29.612970 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdff00ba-2596-47bd-a64a-39dd98aa0788 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:29.613925 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdff00ba-2596-47bd-a64a-39dd98aa0788 admin admin] PATH_INFO: `/v3/projects/f3fdad6f801941b7923342ffe7cb0818` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:29.614634 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdff00ba-2596-47bd-a64a-39dd98aa0788 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f3fdad6f801941b7923342ffe7cb0818)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:29.616613 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdff00ba-2596-47bd-a64a-39dd98aa0788 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:29.666457 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-bdff00ba-2596-47bd-a64a-39dd98aa0788 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f3fdad6f801941b7923342ffe7cb0818', 'request_id': 'req-bdff00ba-2596-47bd-a64a-39dd98aa0788'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:29.673330 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-bdff00ba-2596-47bd-a64a-39dd98aa0788 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f3fdad6f801941b7923342ffe7cb0818', 'request_id': 'req-bdff00ba-2596-47bd-a64a-39dd98aa0788'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:29.679719 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-bdff00ba-2596-47bd-a64a-39dd98aa0788 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f3fdad6f801941b7923342ffe7cb0818', 'request_id': 'req-bdff00ba-2596-47bd-a64a-39dd98aa0788'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:29.690535 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-bdff00ba-2596-47bd-a64a-39dd98aa0788 admin admin] The token cache is being invalidate because project f3fdad6f801941b7923342ffe7cb0818 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81625) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:19:29.690904 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-bdff00ba-2596-47bd-a64a-39dd98aa0788 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:29.691475 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-bdff00ba-2596-47bd-a64a-39dd98aa0788 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:29.692929 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1364/2696] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:19:29 2022] DELETE /identity/v3/projects/f3fdad6f801941b7923342ffe7cb0818 => generated 0 bytes in 100 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:19:29.697815 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f8a084-6c75-4222-8e6b-f28c5c743da9 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:29.698107 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f8a084-6c75-4222-8e6b-f28c5c743da9 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:29.698388 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f8a084-6c75-4222-8e6b-f28c5c743da9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:29.719116 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-80f8a084-6c75-4222-8e6b-f28c5c743da9 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:19:29.720490 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-80f8a084-6c75-4222-8e6b-f28c5c743da9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:19:29.791338 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1333/2697] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:19:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:19:29.793280 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-367beb61-65f1-4ca2-9785-b7f5e9d698fc None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:29.793570 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-367beb61-65f1-4ca2-9785-b7f5e9d698fc None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:29.793841 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-367beb61-65f1-4ca2-9785-b7f5e9d698fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:29.813924 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-367beb61-65f1-4ca2-9785-b7f5e9d698fc None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:19:29.815170 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-367beb61-65f1-4ca2-9785-b7f5e9d698fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:19:29.823743 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1365/2698] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:19:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:19:29.828790 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80f8a084-6c75-4222-8e6b-f28c5c743da9 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:29.838784 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80f8a084-6c75-4222-8e6b-f28c5c743da9 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:29.843804 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a10c6b1-c2f4-4856-ae48-0420064c0f0f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:29.844594 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a10c6b1-c2f4-4856-ae48-0420064c0f0f admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:29.844861 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a10c6b1-c2f4-4856-ae48-0420064c0f0f admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:29.845136 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a10c6b1-c2f4-4856-ae48-0420064c0f0f admin admin] PATH_INFO: `/v3/domains` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:29.845883 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a10c6b1-c2f4-4856-ae48-0420064c0f0f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:19:29.846182 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a10c6b1-c2f4-4856-ae48-0420064c0f0f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:29.847810 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a10c6b1-c2f4-4856-ae48-0420064c0f0f admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:29.859645 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1334/2699] 10.222.0.77 () {58 vars in 1153 bytes} [Tue Aug 30 05:19:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 05:19:29.863966 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-367beb61-65f1-4ca2-9785-b7f5e9d698fc None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:29.873879 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-367beb61-65f1-4ca2-9785-b7f5e9d698fc None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:29.878654 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-089c3a9b-12d8-4c3e-aa52-4201d39df653 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:29.879292 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-089c3a9b-12d8-4c3e-aa52-4201d39df653 admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:29.879569 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-089c3a9b-12d8-4c3e-aa52-4201d39df653 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:29.879950 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-089c3a9b-12d8-4c3e-aa52-4201d39df653 admin admin] PATH_INFO: `/v3/projects` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:29.880586 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-089c3a9b-12d8-4c3e-aa52-4201d39df653 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:29.882526 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-089c3a9b-12d8-4c3e-aa52-4201d39df653 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:29.910027 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1366/2700] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:19:29 2022] POST /identity/v3/projects => generated 385 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 05:19:29.914121 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a10c6b1-c2f4-4856-ae48-0420064c0f0f admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:29.923976 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a10c6b1-c2f4-4856-ae48-0420064c0f0f admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:29.978642 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ececa9df-eccf-40f0-82c3-2d5377063b66 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:29.979291 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ececa9df-eccf-40f0-82c3-2d5377063b66 admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:29.979567 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ececa9df-eccf-40f0-82c3-2d5377063b66 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:29.979835 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ececa9df-eccf-40f0-82c3-2d5377063b66 admin admin] PATH_INFO: `/v3/users` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:29.980451 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ececa9df-eccf-40f0-82c3-2d5377063b66 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:29.982338 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ececa9df-eccf-40f0-82c3-2d5377063b66 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:30.002881 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1335/2701] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:19:29 2022] POST /identity/v3/users => generated 293 bytes in 89 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:19:30.007005 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-089c3a9b-12d8-4c3e-aa52-4201d39df653 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:30.017802 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-089c3a9b-12d8-4c3e-aa52-4201d39df653 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:30.022879 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23e72803-12bb-419e-817f-4f43e99908b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:30.023441 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23e72803-12bb-419e-817f-4f43e99908b8 admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:30.023785 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23e72803-12bb-419e-817f-4f43e99908b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:30.024090 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23e72803-12bb-419e-817f-4f43e99908b8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:30.024758 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23e72803-12bb-419e-817f-4f43e99908b8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:30.026460 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23e72803-12bb-419e-817f-4f43e99908b8 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:30.036342 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1367/2702] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:19:30 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:19:30.040328 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ececa9df-eccf-40f0-82c3-2d5377063b66 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:30.050780 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ececa9df-eccf-40f0-82c3-2d5377063b66 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:30.055378 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1592f63-36b3-453b-8a81-2cfdd28de99f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:30.055956 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1592f63-36b3-453b-8a81-2cfdd28de99f admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:30.056230 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1592f63-36b3-453b-8a81-2cfdd28de99f admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:30.056493 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1592f63-36b3-453b-8a81-2cfdd28de99f admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:30.057116 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1592f63-36b3-453b-8a81-2cfdd28de99f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:30.058690 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1592f63-36b3-453b-8a81-2cfdd28de99f admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:30.069627 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1336/2703] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:19:30 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:19:30.073531 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23e72803-12bb-419e-817f-4f43e99908b8 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:30.083435 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23e72803-12bb-419e-817f-4f43e99908b8 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:30.088012 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5f0f32b-09f3-43a3-b066-b6c2f6ea3cf7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:30.088632 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5f0f32b-09f3-43a3-b066-b6c2f6ea3cf7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:30.088908 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5f0f32b-09f3-43a3-b066-b6c2f6ea3cf7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:30.089176 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5f0f32b-09f3-43a3-b066-b6c2f6ea3cf7 admin admin] PATH_INFO: `/v3/projects/4826d4507ffc4e3288aac00b1409ea46/users/b92e3e3eff1e40d38fdfc68e467367f7/roles/6f03d86980c34c36b44a8d6764ff4fda` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:30.103479 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5f0f32b-09f3-43a3-b066-b6c2f6ea3cf7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4826d4507ffc4e3288aac00b1409ea46, user_id=b92e3e3eff1e40d38fdfc68e467367f7, role_id=6f03d86980c34c36b44a8d6764ff4fda)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:30.106093 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5f0f32b-09f3-43a3-b066-b6c2f6ea3cf7 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:30.109627 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:19:30.123198 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1368/2704] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:19:30 2022] PUT /identity/v3/projects/4826d4507ffc4e3288aac00b1409ea46/users/b92e3e3eff1e40d38fdfc68e467367f7/roles/6f03d86980c34c36b44a8d6764ff4fda => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 05:19:30.128622 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fd40250-7c38-4de5-93e1-4c23e87fbfa3 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:30.128899 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fd40250-7c38-4de5-93e1-4c23e87fbfa3 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:30.129165 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fd40250-7c38-4de5-93e1-4c23e87fbfa3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:30.156128 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-7fd40250-7c38-4de5-93e1-4c23e87fbfa3 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:19:30.157318 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-7fd40250-7c38-4de5-93e1-4c23e87fbfa3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:19:30.203969 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:19:30.217257 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1337/2705] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:19:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:19:30.226799 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5f0f32b-09f3-43a3-b066-b6c2f6ea3cf7 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:30.228646 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-b5f0f32b-09f3-43a3-b066-b6c2f6ea3cf7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:19:30.285924 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5f0f32b-09f3-43a3-b066-b6c2f6ea3cf7 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:30.289494 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e488a6f-2635-4768-b219-a987e0e11070 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:30.290079 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e488a6f-2635-4768-b219-a987e0e11070 service neutron] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:30.290353 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e488a6f-2635-4768-b219-a987e0e11070 service neutron] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:30.290620 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e488a6f-2635-4768-b219-a987e0e11070 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:30.299922 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e488a6f-2635-4768-b219-a987e0e11070 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:30.301374 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e488a6f-2635-4768-b219-a987e0e11070 service neutron] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:30.310473 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1369/2706] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:19:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:19:34.062403 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fd40250-7c38-4de5-93e1-4c23e87fbfa3 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:34.064579 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-7fd40250-7c38-4de5-93e1-4c23e87fbfa3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:19:34.077874 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fd40250-7c38-4de5-93e1-4c23e87fbfa3 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:34.083133 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab04f907-7cc8-43d9-a950-ece5a1ba2807 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:34.084147 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab04f907-7cc8-43d9-a950-ece5a1ba2807 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:34.084674 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab04f907-7cc8-43d9-a950-ece5a1ba2807 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:34.085278 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab04f907-7cc8-43d9-a950-ece5a1ba2807 admin admin] PATH_INFO: `/v3/users/ccb0e0ccc7ec4495a6d5676419103273` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:34.086926 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab04f907-7cc8-43d9-a950-ece5a1ba2807 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ccb0e0ccc7ec4495a6d5676419103273)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:34.088945 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab04f907-7cc8-43d9-a950-ece5a1ba2807 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:34.155829 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-ab04f907-7cc8-43d9-a950-ece5a1ba2807 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ccb0e0ccc7ec4495a6d5676419103273'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:34.156447 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-ab04f907-7cc8-43d9-a950-ece5a1ba2807 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ccb0e0ccc7ec4495a6d5676419103273'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:34.157082 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-ab04f907-7cc8-43d9-a950-ece5a1ba2807 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ccb0e0ccc7ec4495a6d5676419103273'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:34.171436 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-ab04f907-7cc8-43d9-a950-ece5a1ba2807 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ccb0e0ccc7ec4495a6d5676419103273'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:34.182350 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-ab04f907-7cc8-43d9-a950-ece5a1ba2807 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ccb0e0ccc7ec4495a6d5676419103273'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:34.205811 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1338/2707] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:19:34 2022] DELETE /identity/v3/users/ccb0e0ccc7ec4495a6d5676419103273 => generated 0 bytes in 144 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:19:34.211693 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab27f80-e54b-49e3-8393-21dcf7386792 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:34.211693 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab27f80-e54b-49e3-8393-21dcf7386792 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:34.211693 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab27f80-e54b-49e3-8393-21dcf7386792 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:34.231150 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-3ab27f80-e54b-49e3-8393-21dcf7386792 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:19:34.232462 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-3ab27f80-e54b-49e3-8393-21dcf7386792 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:19:34.304946 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1370/2708] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:19:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:19:34.312090 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab04f907-7cc8-43d9-a950-ece5a1ba2807 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:34.314181 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-ab04f907-7cc8-43d9-a950-ece5a1ba2807 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:19:34.383375 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab04f907-7cc8-43d9-a950-ece5a1ba2807 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:34.387295 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f53b019-12a9-47ff-b743-3407fddd8a38 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:34.387970 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f53b019-12a9-47ff-b743-3407fddd8a38 service neutron] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:34.388449 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f53b019-12a9-47ff-b743-3407fddd8a38 service neutron] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:34.388787 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f53b019-12a9-47ff-b743-3407fddd8a38 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:34.411415 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f53b019-12a9-47ff-b743-3407fddd8a38 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:34.413459 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f53b019-12a9-47ff-b743-3407fddd8a38 service neutron] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:34.424914 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1339/2709] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:19:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 114 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:19:34.667065 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9701d48a-f1d2-446a-b737-15bcc341e3da None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:34.667544 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9701d48a-f1d2-446a-b737-15bcc341e3da None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:34.668007 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9701d48a-f1d2-446a-b737-15bcc341e3da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:34.702964 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-9701d48a-f1d2-446a-b737-15bcc341e3da None None] MFA Rules not processed for user `b92e3e3eff1e40d38fdfc68e467367f7`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:19:34.704197 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-9701d48a-f1d2-446a-b737-15bcc341e3da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:19:34.777556 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1371/2710] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:19:34 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:19:34.782723 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f53b019-12a9-47ff-b743-3407fddd8a38 service neutron] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:34.783842 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9701d48a-f1d2-446a-b737-15bcc341e3da None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:34.784149 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-9f53b019-12a9-47ff-b743-3407fddd8a38 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:19:34.786149 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-9701d48a-f1d2-446a-b737-15bcc341e3da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:19:34.802790 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9701d48a-f1d2-446a-b737-15bcc341e3da None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:34.809693 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beace87f-1163-45eb-bd0a-104198dfd688 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:34.810517 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beace87f-1163-45eb-bd0a-104198dfd688 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:34.810957 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beace87f-1163-45eb-bd0a-104198dfd688 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:34.811309 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beace87f-1163-45eb-bd0a-104198dfd688 admin admin] PATH_INFO: `/v3/projects/7b41626669874461afac002a46ee26d7` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:34.812805 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beace87f-1163-45eb-bd0a-104198dfd688 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7b41626669874461afac002a46ee26d7)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:34.813236 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f53b019-12a9-47ff-b743-3407fddd8a38 service neutron] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:34.815102 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beace87f-1163-45eb-bd0a-104198dfd688 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:34.824152 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-974d98fd-ded4-48f0-b8ef-a291ba8ffc92 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:34.824152 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-974d98fd-ded4-48f0-b8ef-a291ba8ffc92 admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:34.824378 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-974d98fd-ded4-48f0-b8ef-a291ba8ffc92 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:34.824484 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-974d98fd-ded4-48f0-b8ef-a291ba8ffc92 admin admin] PATH_INFO: `/v3/projects` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:34.825370 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-974d98fd-ded4-48f0-b8ef-a291ba8ffc92 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:34.829256 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-974d98fd-ded4-48f0-b8ef-a291ba8ffc92 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:34.870138 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=99,keystone:DELETE=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:19:34.876023 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1340/2711] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:19:34 2022] POST /identity/v3/projects => generated 385 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:19:34.880325 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-974d98fd-ded4-48f0-b8ef-a291ba8ffc92 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:34.899170 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-beace87f-1163-45eb-bd0a-104198dfd688 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7b41626669874461afac002a46ee26d7', 'request_id': 'req-beace87f-1163-45eb-bd0a-104198dfd688'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:34.915007 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-beace87f-1163-45eb-bd0a-104198dfd688 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7b41626669874461afac002a46ee26d7', 'request_id': 'req-beace87f-1163-45eb-bd0a-104198dfd688'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:34.923450 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-beace87f-1163-45eb-bd0a-104198dfd688 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7b41626669874461afac002a46ee26d7', 'request_id': 'req-beace87f-1163-45eb-bd0a-104198dfd688'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:34.934750 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-beace87f-1163-45eb-bd0a-104198dfd688 admin admin] The token cache is being invalidate because project 7b41626669874461afac002a46ee26d7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81625) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:19:34.935062 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-beace87f-1163-45eb-bd0a-104198dfd688 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:34.936265 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-beace87f-1163-45eb-bd0a-104198dfd688 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:34.937764 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1372/2712] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:19:34 2022] DELETE /identity/v3/projects/7b41626669874461afac002a46ee26d7 => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:19:34.942040 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:19:34.942883 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beace87f-1163-45eb-bd0a-104198dfd688 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:34.944380 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-beace87f-1163-45eb-bd0a-104198dfd688 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:19:34.960772 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-974d98fd-ded4-48f0-b8ef-a291ba8ffc92 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:35.024128 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beace87f-1163-45eb-bd0a-104198dfd688 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:35.030456 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59dc8e92-5dd7-4de1-a9fa-a4f772bd6fa8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:35.031239 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59dc8e92-5dd7-4de1-a9fa-a4f772bd6fa8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:35.031553 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59dc8e92-5dd7-4de1-a9fa-a4f772bd6fa8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:35.031801 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59dc8e92-5dd7-4de1-a9fa-a4f772bd6fa8 admin admin] PATH_INFO: `/v3/users/972195d72ba0472a84c98c4f897b36c1` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:35.033341 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59dc8e92-5dd7-4de1-a9fa-a4f772bd6fa8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=972195d72ba0472a84c98c4f897b36c1)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:35.035374 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59dc8e92-5dd7-4de1-a9fa-a4f772bd6fa8 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:35.035574 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40063042-eea9-4e3b-bc08-bd03635071c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:35.036318 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40063042-eea9-4e3b-bc08-bd03635071c1 admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:35.036694 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40063042-eea9-4e3b-bc08-bd03635071c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:35.037058 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40063042-eea9-4e3b-bc08-bd03635071c1 admin admin] PATH_INFO: `/v3/users` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:35.037783 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40063042-eea9-4e3b-bc08-bd03635071c1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:35.039805 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40063042-eea9-4e3b-bc08-bd03635071c1 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:35.065888 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1341/2713] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:19:34 2022] POST /identity/v3/users => generated 299 bytes in 186 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:19:35.070514 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40063042-eea9-4e3b-bc08-bd03635071c1 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:35.074703 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-40063042-eea9-4e3b-bc08-bd03635071c1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:19:35.088815 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40063042-eea9-4e3b-bc08-bd03635071c1 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:35.096711 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68408453-5d40-40e4-8841-6db96a1914e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:35.096906 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-59dc8e92-5dd7-4de1-a9fa-a4f772bd6fa8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '972195d72ba0472a84c98c4f897b36c1'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:35.097677 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68408453-5d40-40e4-8841-6db96a1914e6 admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:35.098105 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68408453-5d40-40e4-8841-6db96a1914e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:35.098643 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68408453-5d40-40e4-8841-6db96a1914e6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:35.099607 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68408453-5d40-40e4-8841-6db96a1914e6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:35.101451 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68408453-5d40-40e4-8841-6db96a1914e6 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:35.108436 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-59dc8e92-5dd7-4de1-a9fa-a4f772bd6fa8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '972195d72ba0472a84c98c4f897b36c1'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:35.113966 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1342/2714] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:19:35 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:19:35.118197 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68408453-5d40-40e4-8841-6db96a1914e6 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:35.130557 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68408453-5d40-40e4-8841-6db96a1914e6 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:35.135107 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-59dc8e92-5dd7-4de1-a9fa-a4f772bd6fa8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '972195d72ba0472a84c98c4f897b36c1'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:35.138930 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f9a3050-19ff-4cf8-8cc8-2b996daab23e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:35.140113 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f9a3050-19ff-4cf8-8cc8-2b996daab23e admin admin] REQUEST_METHOD: `PUT` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:35.140391 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f9a3050-19ff-4cf8-8cc8-2b996daab23e admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:35.140802 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f9a3050-19ff-4cf8-8cc8-2b996daab23e admin admin] PATH_INFO: `/v3/projects/b530b1e42d404763b5da765efae99119/users/8cc19d2d467a40cfa9cd51c3e4861b29/roles/9d9d5d2c270146488249b7ed943280db` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:35.144285 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-59dc8e92-5dd7-4de1-a9fa-a4f772bd6fa8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '972195d72ba0472a84c98c4f897b36c1'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:35.145013 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-59dc8e92-5dd7-4de1-a9fa-a4f772bd6fa8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '972195d72ba0472a84c98c4f897b36c1'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:35.155130 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1373/2715] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:19:34 2022] DELETE /identity/v3/users/972195d72ba0472a84c98c4f897b36c1 => generated 0 bytes in 211 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 05:19:35.169117 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f9a3050-19ff-4cf8-8cc8-2b996daab23e admin admin] RBAC: Authorizing `identity:create_grant(project_id=b530b1e42d404763b5da765efae99119, user_id=8cc19d2d467a40cfa9cd51c3e4861b29, role_id=9d9d5d2c270146488249b7ed943280db)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:35.172286 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f9a3050-19ff-4cf8-8cc8-2b996daab23e admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:35.193918 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1343/2716] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:19:35 2022] PUT /identity/v3/projects/b530b1e42d404763b5da765efae99119/users/8cc19d2d467a40cfa9cd51c3e4861b29/roles/9d9d5d2c270146488249b7ed943280db => generated 0 bytes in 73 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:19:35.195197 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59dc8e92-5dd7-4de1-a9fa-a4f772bd6fa8 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:35.196776 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-59dc8e92-5dd7-4de1-a9fa-a4f772bd6fa8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:19:35.237254 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=82,keystone:DELETE=16,keystone:INSERT=2 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:19:35.290643 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59dc8e92-5dd7-4de1-a9fa-a4f772bd6fa8 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:35.379692 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6b10c6a-9751-4ff7-b114-99365c097180 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:35.380494 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6b10c6a-9751-4ff7-b114-99365c097180 admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:35.380861 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6b10c6a-9751-4ff7-b114-99365c097180 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:35.381304 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6b10c6a-9751-4ff7-b114-99365c097180 admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:35.382128 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6b10c6a-9751-4ff7-b114-99365c097180 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:35.383895 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6b10c6a-9751-4ff7-b114-99365c097180 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:35.412393 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1374/2717] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:19:35 2022] GET /identity/v3/roles => generated 1420 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:19:35.416195 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f9a3050-19ff-4cf8-8cc8-2b996daab23e admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:35.433351 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f9a3050-19ff-4cf8-8cc8-2b996daab23e admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:35.439826 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f623449-4519-484e-9120-ed289c027544 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:35.440605 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f623449-4519-484e-9120-ed289c027544 admin admin] REQUEST_METHOD: `PUT` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:35.441155 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f623449-4519-484e-9120-ed289c027544 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:35.441539 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f623449-4519-484e-9120-ed289c027544 admin admin] PATH_INFO: `/v3/projects/b530b1e42d404763b5da765efae99119/users/8cc19d2d467a40cfa9cd51c3e4861b29/roles/9d9d5d2c270146488249b7ed943280db` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:35.443848 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f623449-4519-484e-9120-ed289c027544 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b530b1e42d404763b5da765efae99119, user_id=8cc19d2d467a40cfa9cd51c3e4861b29, role_id=9d9d5d2c270146488249b7ed943280db)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:35.446600 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f623449-4519-484e-9120-ed289c027544 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:35.475484 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1344/2718] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:19:35 2022] PUT /identity/v3/projects/b530b1e42d404763b5da765efae99119/users/8cc19d2d467a40cfa9cd51c3e4861b29/roles/9d9d5d2c270146488249b7ed943280db => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:19:35.532048 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6b10c6a-9751-4ff7-b114-99365c097180 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:35.534077 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-b6b10c6a-9751-4ff7-b114-99365c097180 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:19:35.551301 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6b10c6a-9751-4ff7-b114-99365c097180 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:35.644559 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fe24b37-1ae2-4bf4-bc1a-a01c4424dbe0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:35.645544 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fe24b37-1ae2-4bf4-bc1a-a01c4424dbe0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:35.645972 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fe24b37-1ae2-4bf4-bc1a-a01c4424dbe0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:35.646384 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fe24b37-1ae2-4bf4-bc1a-a01c4424dbe0 admin admin] PATH_INFO: `/v3/projects/8f1968c6ac6643f1910ba73ae236f23c` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:35.647933 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fe24b37-1ae2-4bf4-bc1a-a01c4424dbe0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8f1968c6ac6643f1910ba73ae236f23c)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:35.650093 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fe24b37-1ae2-4bf4-bc1a-a01c4424dbe0 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:35.738422 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-5fe24b37-1ae2-4bf4-bc1a-a01c4424dbe0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8f1968c6ac6643f1910ba73ae236f23c', 'request_id': 'req-5fe24b37-1ae2-4bf4-bc1a-a01c4424dbe0'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:35.753971 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-5fe24b37-1ae2-4bf4-bc1a-a01c4424dbe0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8f1968c6ac6643f1910ba73ae236f23c', 'request_id': 'req-5fe24b37-1ae2-4bf4-bc1a-a01c4424dbe0'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:35.762829 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-5fe24b37-1ae2-4bf4-bc1a-a01c4424dbe0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8f1968c6ac6643f1910ba73ae236f23c', 'request_id': 'req-5fe24b37-1ae2-4bf4-bc1a-a01c4424dbe0'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:35.778288 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-5fe24b37-1ae2-4bf4-bc1a-a01c4424dbe0 admin admin] The token cache is being invalidate because project 8f1968c6ac6643f1910ba73ae236f23c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81625) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:19:35.778933 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-5fe24b37-1ae2-4bf4-bc1a-a01c4424dbe0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:35.780457 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-5fe24b37-1ae2-4bf4-bc1a-a01c4424dbe0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:35.782238 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1375/2719] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:19:35 2022] DELETE /identity/v3/projects/8f1968c6ac6643f1910ba73ae236f23c => generated 0 bytes in 251 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 05:19:35.787232 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e385df9-9476-496d-9b8a-f7de0773f686 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:35.787524 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e385df9-9476-496d-9b8a-f7de0773f686 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:35.787853 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e385df9-9476-496d-9b8a-f7de0773f686 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:35.809785 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-4e385df9-9476-496d-9b8a-f7de0773f686 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:19:35.811112 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-4e385df9-9476-496d-9b8a-f7de0773f686 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:19:35.886675 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1345/2720] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:19:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:19:35.895920 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc93000-28d3-4d63-9245-2a85830aefdb None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:35.896212 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc93000-28d3-4d63-9245-2a85830aefdb None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:35.896487 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc93000-28d3-4d63-9245-2a85830aefdb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:35.907142 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:19:35.923101 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-9dc93000-28d3-4d63-9245-2a85830aefdb None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:19:35.929710 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-9dc93000-28d3-4d63-9245-2a85830aefdb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:19:35.940250 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1376/2721] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:19:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 45 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:19:35.944688 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e385df9-9476-496d-9b8a-f7de0773f686 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:35.955356 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e385df9-9476-496d-9b8a-f7de0773f686 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:35.960217 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c58d8f0b-4710-415a-b64b-a0c79fe2292a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:35.960819 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c58d8f0b-4710-415a-b64b-a0c79fe2292a admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:35.961094 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c58d8f0b-4710-415a-b64b-a0c79fe2292a admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:35.961361 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c58d8f0b-4710-415a-b64b-a0c79fe2292a admin admin] PATH_INFO: `/v3/domains` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:35.962005 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c58d8f0b-4710-415a-b64b-a0c79fe2292a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:19:35.962301 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c58d8f0b-4710-415a-b64b-a0c79fe2292a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:35.963939 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c58d8f0b-4710-415a-b64b-a0c79fe2292a admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:35.984400 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1346/2722] 10.222.0.77 () {58 vars in 1153 bytes} [Tue Aug 30 05:19:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:19:35.989029 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dc93000-28d3-4d63-9245-2a85830aefdb None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:36.001810 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dc93000-28d3-4d63-9245-2a85830aefdb None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:36.012035 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57af4651-e070-4ddb-b423-1eac2d867701 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:36.012913 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57af4651-e070-4ddb-b423-1eac2d867701 admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:36.013264 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57af4651-e070-4ddb-b423-1eac2d867701 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:36.013607 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57af4651-e070-4ddb-b423-1eac2d867701 admin admin] PATH_INFO: `/v3/projects` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:36.014953 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57af4651-e070-4ddb-b423-1eac2d867701 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:36.019040 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57af4651-e070-4ddb-b423-1eac2d867701 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:36.055845 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1377/2723] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:19:35 2022] POST /identity/v3/projects => generated 373 bytes in 67 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:19:36.060055 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c58d8f0b-4710-415a-b64b-a0c79fe2292a admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:36.073455 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c58d8f0b-4710-415a-b64b-a0c79fe2292a admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:36.097978 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:19:36.144150 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f17498c1-dd6e-4f97-b3f6-54d14b18f0dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:36.144770 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f17498c1-dd6e-4f97-b3f6-54d14b18f0dd admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:36.145052 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f17498c1-dd6e-4f97-b3f6-54d14b18f0dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:36.145908 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f17498c1-dd6e-4f97-b3f6-54d14b18f0dd admin admin] PATH_INFO: `/v3/users` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:36.146551 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f17498c1-dd6e-4f97-b3f6-54d14b18f0dd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:36.148515 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f17498c1-dd6e-4f97-b3f6-54d14b18f0dd admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:36.176219 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1347/2724] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:19:36 2022] POST /identity/v3/users => generated 287 bytes in 117 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:19:36.180278 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57af4651-e070-4ddb-b423-1eac2d867701 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:36.195236 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57af4651-e070-4ddb-b423-1eac2d867701 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:36.206667 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a205600b-4162-4aa7-a902-d07967a51061 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:36.207513 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a205600b-4162-4aa7-a902-d07967a51061 admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:36.207787 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a205600b-4162-4aa7-a902-d07967a51061 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:36.208535 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a205600b-4162-4aa7-a902-d07967a51061 admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:36.209187 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a205600b-4162-4aa7-a902-d07967a51061 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:36.210774 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a205600b-4162-4aa7-a902-d07967a51061 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:36.225306 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1378/2725] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:19:36 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:19:36.227628 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f17498c1-dd6e-4f97-b3f6-54d14b18f0dd admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:36.239012 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f17498c1-dd6e-4f97-b3f6-54d14b18f0dd admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:36.245075 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cd09e52-9913-4457-b214-7b5a42133fd4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:36.245697 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cd09e52-9913-4457-b214-7b5a42133fd4 admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:36.245985 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cd09e52-9913-4457-b214-7b5a42133fd4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:36.246243 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cd09e52-9913-4457-b214-7b5a42133fd4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:36.246919 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cd09e52-9913-4457-b214-7b5a42133fd4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:36.248576 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cd09e52-9913-4457-b214-7b5a42133fd4 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:36.265250 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1348/2726] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:19:36 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:19:36.270624 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a205600b-4162-4aa7-a902-d07967a51061 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:36.284386 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a205600b-4162-4aa7-a902-d07967a51061 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:36.289851 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b291e07-e7f2-4d03-9186-9b6a3a167902 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:36.290517 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b291e07-e7f2-4d03-9186-9b6a3a167902 admin admin] REQUEST_METHOD: `PUT` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:36.290796 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b291e07-e7f2-4d03-9186-9b6a3a167902 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:36.291111 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b291e07-e7f2-4d03-9186-9b6a3a167902 admin admin] PATH_INFO: `/v3/projects/89a01e9365474f1b8232535b2008594a/users/fd3d09d079bd4294bee8f1b71734e3b2/roles/6f03d86980c34c36b44a8d6764ff4fda` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:36.316267 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b291e07-e7f2-4d03-9186-9b6a3a167902 admin admin] RBAC: Authorizing `identity:create_grant(project_id=89a01e9365474f1b8232535b2008594a, user_id=fd3d09d079bd4294bee8f1b71734e3b2, role_id=6f03d86980c34c36b44a8d6764ff4fda)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:36.318961 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b291e07-e7f2-4d03-9186-9b6a3a167902 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:36.347326 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1379/2727] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:19:36 2022] PUT /identity/v3/projects/89a01e9365474f1b8232535b2008594a/users/fd3d09d079bd4294bee8f1b71734e3b2/roles/6f03d86980c34c36b44a8d6764ff4fda => generated 0 bytes in 74 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:19:36.348511 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e01be16-c807-47f4-b220-ed3b36dce03b None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:36.348760 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e01be16-c807-47f4-b220-ed3b36dce03b None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:36.349025 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e01be16-c807-47f4-b220-ed3b36dce03b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:36.374048 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-5e01be16-c807-47f4-b220-ed3b36dce03b None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:19:36.375381 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-5e01be16-c807-47f4-b220-ed3b36dce03b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:19:36.461846 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1349/2728] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:19:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 05:19:36.486578 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b291e07-e7f2-4d03-9186-9b6a3a167902 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:36.488762 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-3b291e07-e7f2-4d03-9186-9b6a3a167902 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:19:36.583983 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b291e07-e7f2-4d03-9186-9b6a3a167902 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:36.589079 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcb87b0f-28dc-40f8-ab4b-a6ff541761d7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:36.589734 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcb87b0f-28dc-40f8-ab4b-a6ff541761d7 service neutron] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:36.591022 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcb87b0f-28dc-40f8-ab4b-a6ff541761d7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:36.591328 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcb87b0f-28dc-40f8-ab4b-a6ff541761d7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:36.602809 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcb87b0f-28dc-40f8-ab4b-a6ff541761d7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:36.604502 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcb87b0f-28dc-40f8-ab4b-a6ff541761d7 service neutron] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:36.616375 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1380/2729] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:19:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 131 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:19:38.163838 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e29c9f5-c7e3-46c5-998f-6961abec274b None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:38.164429 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e29c9f5-c7e3-46c5-998f-6961abec274b None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:38.164634 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e29c9f5-c7e3-46c5-998f-6961abec274b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:38.231941 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-3e29c9f5-c7e3-46c5-998f-6961abec274b None None] MFA Rules not processed for user `fd3d09d079bd4294bee8f1b71734e3b2`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:19:38.233471 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-3e29c9f5-c7e3-46c5-998f-6961abec274b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:19:38.425453 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1350/2730] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:19:38 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 263 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:19:38.429853 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcb87b0f-28dc-40f8-ab4b-a6ff541761d7 service neutron] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:38.453251 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcb87b0f-28dc-40f8-ab4b-a6ff541761d7 service neutron] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:38.474389 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad624c8a-a530-4b1f-8936-d899dd8c1bf2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:38.476766 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad624c8a-a530-4b1f-8936-d899dd8c1bf2 admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:38.477210 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad624c8a-a530-4b1f-8936-d899dd8c1bf2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:38.478644 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad624c8a-a530-4b1f-8936-d899dd8c1bf2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:38.479986 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad624c8a-a530-4b1f-8936-d899dd8c1bf2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:38.483396 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad624c8a-a530-4b1f-8936-d899dd8c1bf2 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:38.543904 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1381/2731] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:19:38 2022] POST /identity/v3/projects => generated 373 bytes in 114 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:19:38.549529 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e29c9f5-c7e3-46c5-998f-6961abec274b None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:38.574140 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e29c9f5-c7e3-46c5-998f-6961abec274b None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:38.626632 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:19:38.669919 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30a51dae-6380-4f51-8348-946191bae66c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:38.671711 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30a51dae-6380-4f51-8348-946191bae66c admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:38.672151 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30a51dae-6380-4f51-8348-946191bae66c admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:38.672597 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30a51dae-6380-4f51-8348-946191bae66c admin admin] PATH_INFO: `/v3/users` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:38.673463 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30a51dae-6380-4f51-8348-946191bae66c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:38.675866 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30a51dae-6380-4f51-8348-946191bae66c admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:38.703932 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1351/2732] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:19:38 2022] POST /identity/v3/users => generated 293 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:19:38.707778 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad624c8a-a530-4b1f-8936-d899dd8c1bf2 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:38.726563 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad624c8a-a530-4b1f-8936-d899dd8c1bf2 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:38.734853 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e814e3c-55ad-4ee7-a083-658cc1ee3feb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:38.735918 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e814e3c-55ad-4ee7-a083-658cc1ee3feb admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:38.736449 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e814e3c-55ad-4ee7-a083-658cc1ee3feb admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:38.736988 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e814e3c-55ad-4ee7-a083-658cc1ee3feb admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:38.737901 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e814e3c-55ad-4ee7-a083-658cc1ee3feb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:38.740224 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e814e3c-55ad-4ee7-a083-658cc1ee3feb admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:38.757427 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1382/2733] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:19:38 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:19:38.762347 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30a51dae-6380-4f51-8348-946191bae66c admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:38.787458 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30a51dae-6380-4f51-8348-946191bae66c admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:38.796723 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-734335d8-eac6-4782-81ac-6aaa86069fa1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:38.797733 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-734335d8-eac6-4782-81ac-6aaa86069fa1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:38.797905 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-734335d8-eac6-4782-81ac-6aaa86069fa1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:38.798281 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-734335d8-eac6-4782-81ac-6aaa86069fa1 admin admin] PATH_INFO: `/v3/projects/b7cb5a2a89cb44a99d0281a2212905c8/users/6d6c63b03af34238923e78d45dcb11a4/roles/9d9d5d2c270146488249b7ed943280db` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:38.842536 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-734335d8-eac6-4782-81ac-6aaa86069fa1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b7cb5a2a89cb44a99d0281a2212905c8, user_id=6d6c63b03af34238923e78d45dcb11a4, role_id=9d9d5d2c270146488249b7ed943280db)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:38.848147 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-734335d8-eac6-4782-81ac-6aaa86069fa1 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:38.875197 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1352/2734] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:19:38 2022] PUT /identity/v3/projects/b7cb5a2a89cb44a99d0281a2212905c8/users/6d6c63b03af34238923e78d45dcb11a4/roles/9d9d5d2c270146488249b7ed943280db => generated 0 bytes in 114 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:19:38.879054 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e814e3c-55ad-4ee7-a083-658cc1ee3feb admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:38.889693 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e814e3c-55ad-4ee7-a083-658cc1ee3feb admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:39.049123 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5382bb9-9442-4b4f-bc1c-ce8b23a03b64 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:39.050541 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5382bb9-9442-4b4f-bc1c-ce8b23a03b64 admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:39.050929 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5382bb9-9442-4b4f-bc1c-ce8b23a03b64 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:39.051435 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5382bb9-9442-4b4f-bc1c-ce8b23a03b64 admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:39.060339 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5382bb9-9442-4b4f-bc1c-ce8b23a03b64 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:39.067310 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5382bb9-9442-4b4f-bc1c-ce8b23a03b64 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:39.091490 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1383/2735] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:19:38 2022] GET /identity/v3/roles => generated 1420 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:19:39.096239 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-734335d8-eac6-4782-81ac-6aaa86069fa1 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:39.115635 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-734335d8-eac6-4782-81ac-6aaa86069fa1 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:39.127002 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cb3539c-8861-4f6c-8b05-d0a310d1567e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:39.127789 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cb3539c-8861-4f6c-8b05-d0a310d1567e admin admin] REQUEST_METHOD: `PUT` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:39.128171 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cb3539c-8861-4f6c-8b05-d0a310d1567e admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:39.128545 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cb3539c-8861-4f6c-8b05-d0a310d1567e admin admin] PATH_INFO: `/v3/projects/b7cb5a2a89cb44a99d0281a2212905c8/users/6d6c63b03af34238923e78d45dcb11a4/roles/9d9d5d2c270146488249b7ed943280db` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:39.136732 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cb3539c-8861-4f6c-8b05-d0a310d1567e admin admin] RBAC: Authorizing `identity:create_grant(project_id=b7cb5a2a89cb44a99d0281a2212905c8, user_id=6d6c63b03af34238923e78d45dcb11a4, role_id=9d9d5d2c270146488249b7ed943280db)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:39.140513 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cb3539c-8861-4f6c-8b05-d0a310d1567e admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:39.179171 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1353/2736] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:19:39 2022] PUT /identity/v3/projects/b7cb5a2a89cb44a99d0281a2212905c8/users/6d6c63b03af34238923e78d45dcb11a4/roles/9d9d5d2c270146488249b7ed943280db => generated 0 bytes in 80 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:19:40.774310 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-470fb985-f7b2-4245-b792-0eb0ea09320d None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:40.774921 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-470fb985-f7b2-4245-b792-0eb0ea09320d None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:40.774921 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-470fb985-f7b2-4245-b792-0eb0ea09320d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:40.796518 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:19:40.838163 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-470fb985-f7b2-4245-b792-0eb0ea09320d None None] MFA Rules not processed for user `6d6c63b03af34238923e78d45dcb11a4`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:19:40.841257 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-470fb985-f7b2-4245-b792-0eb0ea09320d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:19:40.972979 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1384/2737] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:19:40 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 200 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:19:40.985333 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cb3539c-8861-4f6c-8b05-d0a310d1567e admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:40.987129 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-5cb3539c-8861-4f6c-8b05-d0a310d1567e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:19:41.100368 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cb3539c-8861-4f6c-8b05-d0a310d1567e admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:41.105038 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74a1c6d6-86c7-4c99-a651-bc4cad1115c4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '019a79b2affe478b8a30d1c538d0415c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:41.106048 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74a1c6d6-86c7-4c99-a651-bc4cad1115c4 service cinder] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:41.106505 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74a1c6d6-86c7-4c99-a651-bc4cad1115c4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:41.107199 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74a1c6d6-86c7-4c99-a651-bc4cad1115c4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:41.200918 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74a1c6d6-86c7-4c99-a651-bc4cad1115c4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:41.204427 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74a1c6d6-86c7-4c99-a651-bc4cad1115c4 service cinder] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:41.229604 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1354/2738] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:19:40 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 244 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:19:41.413086 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff143918-ada2-49f5-ad98-e8980f7af2be None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:41.413378 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff143918-ada2-49f5-ad98-e8980f7af2be None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:41.413674 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff143918-ada2-49f5-ad98-e8980f7af2be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:41.457957 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-ff143918-ada2-49f5-ad98-e8980f7af2be None None] MFA Rules not processed for user `8cc19d2d467a40cfa9cd51c3e4861b29`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:19:41.460032 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-ff143918-ada2-49f5-ad98-e8980f7af2be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:19:41.543765 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1385/2739] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:19:41 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:19:41.559544 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74a1c6d6-86c7-4c99-a651-bc4cad1115c4 service cinder] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:41.561485 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-74a1c6d6-86c7-4c99-a651-bc4cad1115c4 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:19:41.638347 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:19:41.661478 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74a1c6d6-86c7-4c99-a651-bc4cad1115c4 service cinder] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:41.665441 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec6919f4-5807-468e-9619-062bef6e457f service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a002ac0bf240a891ac7304178d3f0e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:41.666589 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec6919f4-5807-468e-9619-062bef6e457f service glance] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:41.667011 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec6919f4-5807-468e-9619-062bef6e457f service glance] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:41.667448 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec6919f4-5807-468e-9619-062bef6e457f service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:41.669473 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-ec6919f4-5807-468e-9619-062bef6e457f service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:19:41.745347 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec6919f4-5807-468e-9619-062bef6e457f service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:41.746990 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec6919f4-5807-468e-9619-062bef6e457f service glance] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:41.756012 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1355/2740] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:19:41 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 197 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:19:41.897465 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9397480f-8958-4fe7-99f3-553784c681df None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:41.897931 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9397480f-8958-4fe7-99f3-553784c681df None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:41.898168 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9397480f-8958-4fe7-99f3-553784c681df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:41.919440 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-9397480f-8958-4fe7-99f3-553784c681df None None] MFA Rules not processed for user `fe51dbec91624ed1ab00b441e29a3dd5`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:19:41.921449 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-9397480f-8958-4fe7-99f3-553784c681df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:19:41.983270 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1386/2741] 10.222.0.77 () {58 vars in 1009 bytes} [Tue Aug 30 05:19:41 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:19:42.035964 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec6919f4-5807-468e-9619-062bef6e457f service glance] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:42.037460 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-ec6919f4-5807-468e-9619-062bef6e457f service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:19:42.110780 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec6919f4-5807-468e-9619-062bef6e457f service glance] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:42.115807 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39901d8c-3808-4211-83ac-e70258539d8d service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a7f7afc3d44ec788c05d005e52a1b8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:42.116446 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39901d8c-3808-4211-83ac-e70258539d8d service swift] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:42.116724 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39901d8c-3808-4211-83ac-e70258539d8d service swift] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:42.116991 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39901d8c-3808-4211-83ac-e70258539d8d service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:42.125901 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39901d8c-3808-4211-83ac-e70258539d8d service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:42.127546 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39901d8c-3808-4211-83ac-e70258539d8d service swift] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:42.136354 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1356/2742] 10.222.0.77 () {60 vars in 1384 bytes} [Tue Aug 30 05:19:42 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 30 05:19:42.720516 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9397480f-8958-4fe7-99f3-553784c681df None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:42.721854 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-9397480f-8958-4fe7-99f3-553784c681df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:19:42.735678 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9397480f-8958-4fe7-99f3-553784c681df None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:42.739799 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91e4ab58-0e2b-4d98-a2e0-d96ceaad8cbc service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '019a79b2affe478b8a30d1c538d0415c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:42.740418 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e4ab58-0e2b-4d98-a2e0-d96ceaad8cbc service cinder] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:42.740691 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e4ab58-0e2b-4d98-a2e0-d96ceaad8cbc service cinder] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:42.740960 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e4ab58-0e2b-4d98-a2e0-d96ceaad8cbc service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:42.756704 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91e4ab58-0e2b-4d98-a2e0-d96ceaad8cbc service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:42.758161 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91e4ab58-0e2b-4d98-a2e0-d96ceaad8cbc service cinder] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:42.770285 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1387/2743] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:19:42 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 50 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:19:51.004901 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96be84f4-ff03-446e-89f2-ddf418e1d8e0 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:51.005454 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96be84f4-ff03-446e-89f2-ddf418e1d8e0 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:51.005454 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96be84f4-ff03-446e-89f2-ddf418e1d8e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:51.025350 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-96be84f4-ff03-446e-89f2-ddf418e1d8e0 None None] MFA Rules not processed for user `b92e3e3eff1e40d38fdfc68e467367f7`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:19:51.026546 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-96be84f4-ff03-446e-89f2-ddf418e1d8e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:19:51.035129 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1357/2744] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:19:51 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:19:52.768971 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=96 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:19:52.941424 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c35b65-59d8-41ff-89d5-744f52627af7 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:52.941707 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c35b65-59d8-41ff-89d5-744f52627af7 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:52.942336 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c35b65-59d8-41ff-89d5-744f52627af7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:52.972831 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-16c35b65-59d8-41ff-89d5-744f52627af7 None None] MFA Rules not processed for user `fe51dbec91624ed1ab00b441e29a3dd5`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:19:52.974041 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-16c35b65-59d8-41ff-89d5-744f52627af7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:19:52.981189 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1388/2745] 10.222.0.77 () {58 vars in 1009 bytes} [Tue Aug 30 05:19:52 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:19:52.994125 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96be84f4-ff03-446e-89f2-ddf418e1d8e0 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:53.031694 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96be84f4-ff03-446e-89f2-ddf418e1d8e0 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:53.036769 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62398ff0-83db-4047-a5b7-c02990847c91 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a7f7afc3d44ec788c05d005e52a1b8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:53.037372 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62398ff0-83db-4047-a5b7-c02990847c91 service swift] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:53.037650 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62398ff0-83db-4047-a5b7-c02990847c91 service swift] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:53.037927 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62398ff0-83db-4047-a5b7-c02990847c91 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:53.047331 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62398ff0-83db-4047-a5b7-c02990847c91 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:53.048721 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62398ff0-83db-4047-a5b7-c02990847c91 service swift] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:53.057248 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1358/2746] 10.222.0.77 () {60 vars in 1384 bytes} [Tue Aug 30 05:19:52 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 64 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 30 05:19:56.391367 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16c35b65-59d8-41ff-89d5-744f52627af7 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:56.393596 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-16c35b65-59d8-41ff-89d5-744f52627af7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:19:56.471033 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62398ff0-83db-4047-a5b7-c02990847c91 service swift] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:56.549132 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=100 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:19:56.583057 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16c35b65-59d8-41ff-89d5-744f52627af7 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:56.588421 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85442512-53c4-454c-a03f-b74d07dc7df3 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1972959402704b89925001cd8d90cb0e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:56.589246 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85442512-53c4-454c-a03f-b74d07dc7df3 service placement] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:56.589590 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85442512-53c4-454c-a03f-b74d07dc7df3 service placement] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:56.590186 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85442512-53c4-454c-a03f-b74d07dc7df3 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:56.592202 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-85442512-53c4-454c-a03f-b74d07dc7df3 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:19:56.593534 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62398ff0-83db-4047-a5b7-c02990847c91 service swift] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:56.598461 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5bd1526-ee23-45c9-8adc-0c264cbbab9f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:56.599182 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5bd1526-ee23-45c9-8adc-0c264cbbab9f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:56.599567 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5bd1526-ee23-45c9-8adc-0c264cbbab9f admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:56.599946 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5bd1526-ee23-45c9-8adc-0c264cbbab9f admin admin] PATH_INFO: `/v3/users/fd3d09d079bd4294bee8f1b71734e3b2` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:56.601178 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5bd1526-ee23-45c9-8adc-0c264cbbab9f admin admin] RBAC: Authorizing `identity:delete_user(user_id=fd3d09d079bd4294bee8f1b71734e3b2)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:56.603955 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5bd1526-ee23-45c9-8adc-0c264cbbab9f admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:56.606187 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85442512-53c4-454c-a03f-b74d07dc7df3 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:56.608533 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85442512-53c4-454c-a03f-b74d07dc7df3 service placement] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:56.625616 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1389/2747] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:19:56 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 242 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:19:56.672306 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-b5bd1526-ee23-45c9-8adc-0c264cbbab9f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fd3d09d079bd4294bee8f1b71734e3b2'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:56.672978 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-b5bd1526-ee23-45c9-8adc-0c264cbbab9f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fd3d09d079bd4294bee8f1b71734e3b2'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:56.673600 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-b5bd1526-ee23-45c9-8adc-0c264cbbab9f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fd3d09d079bd4294bee8f1b71734e3b2'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:56.687024 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-b5bd1526-ee23-45c9-8adc-0c264cbbab9f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fd3d09d079bd4294bee8f1b71734e3b2'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:56.694659 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-b5bd1526-ee23-45c9-8adc-0c264cbbab9f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fd3d09d079bd4294bee8f1b71734e3b2'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:56.707937 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1359/2748] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:19:56 2022] DELETE /identity/v3/users/fd3d09d079bd4294bee8f1b71734e3b2 => generated 0 bytes in 238 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 05:19:56.888172 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85442512-53c4-454c-a03f-b74d07dc7df3 service placement] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:56.918046 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85442512-53c4-454c-a03f-b74d07dc7df3 service placement] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:56.983585 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cba20357-6031-4a69-9bea-ea9d122aa370 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:56.984316 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cba20357-6031-4a69-9bea-ea9d122aa370 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:56.984740 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cba20357-6031-4a69-9bea-ea9d122aa370 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:56.985023 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cba20357-6031-4a69-9bea-ea9d122aa370 admin admin] PATH_INFO: `/v3/projects/89a01e9365474f1b8232535b2008594a` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:56.986196 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cba20357-6031-4a69-9bea-ea9d122aa370 admin admin] RBAC: Authorizing `identity:delete_project(project_id=89a01e9365474f1b8232535b2008594a)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:56.988397 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cba20357-6031-4a69-9bea-ea9d122aa370 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:57.036151 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:19:57.047165 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-cba20357-6031-4a69-9bea-ea9d122aa370 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '89a01e9365474f1b8232535b2008594a', 'request_id': 'req-cba20357-6031-4a69-9bea-ea9d122aa370'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:57.057093 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-cba20357-6031-4a69-9bea-ea9d122aa370 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '89a01e9365474f1b8232535b2008594a', 'request_id': 'req-cba20357-6031-4a69-9bea-ea9d122aa370'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:57.066276 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-cba20357-6031-4a69-9bea-ea9d122aa370 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '89a01e9365474f1b8232535b2008594a', 'request_id': 'req-cba20357-6031-4a69-9bea-ea9d122aa370'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:57.090968 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-cba20357-6031-4a69-9bea-ea9d122aa370 admin admin] The token cache is being invalidate because project 89a01e9365474f1b8232535b2008594a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81625) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:19:57.091277 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-cba20357-6031-4a69-9bea-ea9d122aa370 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:57.092629 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-cba20357-6031-4a69-9bea-ea9d122aa370 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:57.094409 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1390/2749] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:19:56 2022] DELETE /identity/v3/projects/89a01e9365474f1b8232535b2008594a => generated 0 bytes in 207 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:19:57.098276 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5bd1526-ee23-45c9-8adc-0c264cbbab9f admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:57.099849 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-b5bd1526-ee23-45c9-8adc-0c264cbbab9f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:19:57.113615 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5bd1526-ee23-45c9-8adc-0c264cbbab9f admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:57.174312 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c2b71dc-5bf1-4c6d-bbe8-bac7c192646a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:57.174908 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c2b71dc-5bf1-4c6d-bbe8-bac7c192646a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:57.175410 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c2b71dc-5bf1-4c6d-bbe8-bac7c192646a admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:57.175685 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c2b71dc-5bf1-4c6d-bbe8-bac7c192646a admin admin] PATH_INFO: `/v3/users/6d6c63b03af34238923e78d45dcb11a4` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:57.176836 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c2b71dc-5bf1-4c6d-bbe8-bac7c192646a admin admin] RBAC: Authorizing `identity:delete_user(user_id=6d6c63b03af34238923e78d45dcb11a4)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:57.179069 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c2b71dc-5bf1-4c6d-bbe8-bac7c192646a admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:57.222870 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=83,keystone:DELETE=16,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:19:57.228509 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-0c2b71dc-5bf1-4c6d-bbe8-bac7c192646a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6d6c63b03af34238923e78d45dcb11a4'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:57.235567 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-0c2b71dc-5bf1-4c6d-bbe8-bac7c192646a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6d6c63b03af34238923e78d45dcb11a4'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:57.236558 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-0c2b71dc-5bf1-4c6d-bbe8-bac7c192646a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6d6c63b03af34238923e78d45dcb11a4'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:57.250915 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-0c2b71dc-5bf1-4c6d-bbe8-bac7c192646a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6d6c63b03af34238923e78d45dcb11a4'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:57.258506 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-0c2b71dc-5bf1-4c6d-bbe8-bac7c192646a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6d6c63b03af34238923e78d45dcb11a4'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:57.272237 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1360/2750] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:19:57 2022] DELETE /identity/v3/users/6d6c63b03af34238923e78d45dcb11a4 => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:19:57.469598 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cba20357-6031-4a69-9bea-ea9d122aa370 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:57.474139 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-cba20357-6031-4a69-9bea-ea9d122aa370 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:19:57.497249 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cba20357-6031-4a69-9bea-ea9d122aa370 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:57.592416 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0989ede-362b-49ee-b82e-66c22dce7502 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:57.593250 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0989ede-362b-49ee-b82e-66c22dce7502 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:57.593703 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0989ede-362b-49ee-b82e-66c22dce7502 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:57.593990 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0989ede-362b-49ee-b82e-66c22dce7502 admin admin] PATH_INFO: `/v3/projects/b7cb5a2a89cb44a99d0281a2212905c8` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:57.595208 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0989ede-362b-49ee-b82e-66c22dce7502 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b7cb5a2a89cb44a99d0281a2212905c8)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:57.597584 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0989ede-362b-49ee-b82e-66c22dce7502 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:57.668158 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-d0989ede-362b-49ee-b82e-66c22dce7502 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b7cb5a2a89cb44a99d0281a2212905c8', 'request_id': 'req-d0989ede-362b-49ee-b82e-66c22dce7502'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:57.678216 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-d0989ede-362b-49ee-b82e-66c22dce7502 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b7cb5a2a89cb44a99d0281a2212905c8', 'request_id': 'req-d0989ede-362b-49ee-b82e-66c22dce7502'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:57.687111 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-d0989ede-362b-49ee-b82e-66c22dce7502 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b7cb5a2a89cb44a99d0281a2212905c8', 'request_id': 'req-d0989ede-362b-49ee-b82e-66c22dce7502'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:57.699519 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-d0989ede-362b-49ee-b82e-66c22dce7502 admin admin] The token cache is being invalidate because project b7cb5a2a89cb44a99d0281a2212905c8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81625) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:19:57.699896 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-d0989ede-362b-49ee-b82e-66c22dce7502 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:57.700627 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-d0989ede-362b-49ee-b82e-66c22dce7502 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:19:57.702799 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1391/2751] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:19:57 2022] DELETE /identity/v3/projects/b7cb5a2a89cb44a99d0281a2212905c8 => generated 0 bytes in 234 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:19:57.707940 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5f0a1f3-ef2d-409c-b188-62974776d5da None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:57.708434 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5f0a1f3-ef2d-409c-b188-62974776d5da None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:57.708653 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5f0a1f3-ef2d-409c-b188-62974776d5da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:57.771882 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-b5f0a1f3-ef2d-409c-b188-62974776d5da None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:19:57.773143 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-b5f0a1f3-ef2d-409c-b188-62974776d5da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:19:57.837594 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1361/2752] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:19:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:19:57.845778 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a7be838-0098-4e92-b5ec-391452d0a2f3 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:57.846360 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a7be838-0098-4e92-b5ec-391452d0a2f3 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:57.846735 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a7be838-0098-4e92-b5ec-391452d0a2f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:57.868954 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-1a7be838-0098-4e92-b5ec-391452d0a2f3 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:19:57.870311 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-1a7be838-0098-4e92-b5ec-391452d0a2f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:19:57.879035 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1392/2753] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:19:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:19:57.884009 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5f0a1f3-ef2d-409c-b188-62974776d5da None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:57.893978 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5f0a1f3-ef2d-409c-b188-62974776d5da None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:57.898614 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4ac80da-286d-4638-a526-d55574d56dcc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:57.899268 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4ac80da-286d-4638-a526-d55574d56dcc admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:57.899545 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4ac80da-286d-4638-a526-d55574d56dcc admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:57.899810 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4ac80da-286d-4638-a526-d55574d56dcc admin admin] PATH_INFO: `/v3/domains` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:57.900483 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4ac80da-286d-4638-a526-d55574d56dcc admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:19:57.900784 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4ac80da-286d-4638-a526-d55574d56dcc admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:57.902415 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4ac80da-286d-4638-a526-d55574d56dcc admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:57.914791 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1362/2754] 10.222.0.77 () {58 vars in 1153 bytes} [Tue Aug 30 05:19:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:19:57.919238 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a7be838-0098-4e92-b5ec-391452d0a2f3 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:57.929464 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a7be838-0098-4e92-b5ec-391452d0a2f3 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:57.934680 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4bd15f9-0c9b-436a-98f0-b0e88b9fd8c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:57.935428 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4bd15f9-0c9b-436a-98f0-b0e88b9fd8c1 admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:57.936549 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4bd15f9-0c9b-436a-98f0-b0e88b9fd8c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:57.937049 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4bd15f9-0c9b-436a-98f0-b0e88b9fd8c1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:57.937793 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4bd15f9-0c9b-436a-98f0-b0e88b9fd8c1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:57.939803 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4bd15f9-0c9b-436a-98f0-b0e88b9fd8c1 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:57.970938 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1393/2755] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:19:57 2022] POST /identity/v3/projects => generated 377 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 05:19:57.972304 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4ac80da-286d-4638-a526-d55574d56dcc admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:57.983434 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4ac80da-286d-4638-a526-d55574d56dcc admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:58.045514 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf3b20b5-8490-4a93-9342-df45fd647135 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:58.046114 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf3b20b5-8490-4a93-9342-df45fd647135 admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:58.046397 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf3b20b5-8490-4a93-9342-df45fd647135 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:58.046693 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf3b20b5-8490-4a93-9342-df45fd647135 admin admin] PATH_INFO: `/v3/users` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:58.047369 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf3b20b5-8490-4a93-9342-df45fd647135 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:58.049292 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf3b20b5-8490-4a93-9342-df45fd647135 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:58.074422 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1363/2756] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:19:57 2022] POST /identity/v3/users => generated 289 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:19:58.075728 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4bd15f9-0c9b-436a-98f0-b0e88b9fd8c1 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:58.087437 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4bd15f9-0c9b-436a-98f0-b0e88b9fd8c1 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:58.092855 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-881a41a6-01d0-4168-8bd6-7b84da687166 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:58.093452 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-881a41a6-01d0-4168-8bd6-7b84da687166 admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:58.093727 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-881a41a6-01d0-4168-8bd6-7b84da687166 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:58.093992 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-881a41a6-01d0-4168-8bd6-7b84da687166 admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:58.094627 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-881a41a6-01d0-4168-8bd6-7b84da687166 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:58.096252 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-881a41a6-01d0-4168-8bd6-7b84da687166 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:58.107102 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1394/2757] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:19:58 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:19:58.111294 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf3b20b5-8490-4a93-9342-df45fd647135 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:58.118570 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=92,keystone:DELETE=4,keystone:INSERT=4 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:19:58.122676 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf3b20b5-8490-4a93-9342-df45fd647135 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:58.129002 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4e7a2da-ba46-480e-9785-03240d53ecfc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:58.129729 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4e7a2da-ba46-480e-9785-03240d53ecfc admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:58.130383 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4e7a2da-ba46-480e-9785-03240d53ecfc admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:58.130976 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4e7a2da-ba46-480e-9785-03240d53ecfc admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:58.131623 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4e7a2da-ba46-480e-9785-03240d53ecfc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:58.133291 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4e7a2da-ba46-480e-9785-03240d53ecfc admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:58.148632 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1364/2758] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:19:58 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 05:19:58.149872 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-881a41a6-01d0-4168-8bd6-7b84da687166 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:58.156282 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=86,keystone:DELETE=11,keystone:INSERT=3 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:19:58.160759 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-881a41a6-01d0-4168-8bd6-7b84da687166 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:58.166942 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f07268bc-a80a-4372-82aa-a114e2655991 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:58.167775 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f07268bc-a80a-4372-82aa-a114e2655991 admin admin] REQUEST_METHOD: `PUT` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:58.172402 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f07268bc-a80a-4372-82aa-a114e2655991 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:58.173421 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f07268bc-a80a-4372-82aa-a114e2655991 admin admin] PATH_INFO: `/v3/projects/a2d6bd5a77fd46e696cfa95aa85def3a/users/2087165506e54a42aaa5c41478405824/roles/6f03d86980c34c36b44a8d6764ff4fda` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:58.192108 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f07268bc-a80a-4372-82aa-a114e2655991 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a2d6bd5a77fd46e696cfa95aa85def3a, user_id=2087165506e54a42aaa5c41478405824, role_id=6f03d86980c34c36b44a8d6764ff4fda)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:58.195603 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f07268bc-a80a-4372-82aa-a114e2655991 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:58.213028 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1395/2759] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:19:58 2022] PUT /identity/v3/projects/a2d6bd5a77fd46e696cfa95aa85def3a/users/2087165506e54a42aaa5c41478405824/roles/6f03d86980c34c36b44a8d6764ff4fda => generated 0 bytes in 63 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:19:58.223818 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b40795ea-d7e8-45cb-8e51-ffebff3247ec None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:58.224141 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b40795ea-d7e8-45cb-8e51-ffebff3247ec None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:58.224310 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b40795ea-d7e8-45cb-8e51-ffebff3247ec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:58.265609 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-b40795ea-d7e8-45cb-8e51-ffebff3247ec None None] MFA Rules not processed for user `2087165506e54a42aaa5c41478405824`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:19:58.271448 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-b40795ea-d7e8-45cb-8e51-ffebff3247ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:19:58.347684 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1365/2760] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:19:58 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:19:58.355478 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f07268bc-a80a-4372-82aa-a114e2655991 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:58.360742 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-f07268bc-a80a-4372-82aa-a114e2655991 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:19:58.440678 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f07268bc-a80a-4372-82aa-a114e2655991 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:58.445921 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b19ebebd-de2e-462f-b8b1-d9f7647f17de service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '019a79b2affe478b8a30d1c538d0415c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:58.446318 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b19ebebd-de2e-462f-b8b1-d9f7647f17de service cinder] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:58.446959 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b19ebebd-de2e-462f-b8b1-d9f7647f17de service cinder] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:58.446959 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b19ebebd-de2e-462f-b8b1-d9f7647f17de service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:58.459778 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b19ebebd-de2e-462f-b8b1-d9f7647f17de service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:19:58.461904 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b19ebebd-de2e-462f-b8b1-d9f7647f17de service cinder] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:19:58.474534 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1396/2761] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:19:58 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 120 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:19:59.915024 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b40795ea-d7e8-45cb-8e51-ffebff3247ec None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:19:59.916629 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-b40795ea-d7e8-45cb-8e51-ffebff3247ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:19:59.936142 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b40795ea-d7e8-45cb-8e51-ffebff3247ec None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:19:59.948613 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6990b656-eae8-41ce-bef8-ede3be500b01 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '019a79b2affe478b8a30d1c538d0415c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:19:59.949440 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6990b656-eae8-41ce-bef8-ede3be500b01 service cinder] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:19:59.949785 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6990b656-eae8-41ce-bef8-ede3be500b01 service cinder] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:19:59.950124 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6990b656-eae8-41ce-bef8-ede3be500b01 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:19:59.953626 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-6990b656-eae8-41ce-bef8-ede3be500b01 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:20:00.055256 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6990b656-eae8-41ce-bef8-ede3be500b01 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:20:00.057526 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6990b656-eae8-41ce-bef8-ede3be500b01 service cinder] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:20:00.083836 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1366/2762] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:19:59 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 169 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:20:08.468770 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=34,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:20:10.064700 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=63 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:20:10.220792 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b19ebebd-de2e-462f-b8b1-d9f7647f17de service cinder] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:20:10.222323 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-b19ebebd-de2e-462f-b8b1-d9f7647f17de service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:20:10.291931 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b19ebebd-de2e-462f-b8b1-d9f7647f17de service cinder] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:20:10.297090 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf0e5c94-2167-4feb-997b-094b38f7fb28 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1972959402704b89925001cd8d90cb0e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:20:10.297680 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf0e5c94-2167-4feb-997b-094b38f7fb28 service placement] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:20:10.297954 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf0e5c94-2167-4feb-997b-094b38f7fb28 service placement] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:20:10.298225 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf0e5c94-2167-4feb-997b-094b38f7fb28 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:20:10.300025 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-bf0e5c94-2167-4feb-997b-094b38f7fb28 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:20:10.311362 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf0e5c94-2167-4feb-997b-094b38f7fb28 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:20:10.312919 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf0e5c94-2167-4feb-997b-094b38f7fb28 service placement] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:20:10.323087 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1397/2763] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:20:10 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:20:12.187444 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c70189d8-6891-44cb-a8d6-79f08bbfd26d None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:20:12.188039 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c70189d8-6891-44cb-a8d6-79f08bbfd26d None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:20:12.188039 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c70189d8-6891-44cb-a8d6-79f08bbfd26d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:20:12.208293 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-c70189d8-6891-44cb-a8d6-79f08bbfd26d None None] MFA Rules not processed for user `9256fd43e5124fb68d4e5453584d1e8d`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:20:12.209641 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-c70189d8-6891-44cb-a8d6-79f08bbfd26d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:20:12.218230 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1367/2764] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:20:12 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:20:20.318897 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=32 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:20:22.202271 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=3 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:20:22.671723 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf0e5c94-2167-4feb-997b-094b38f7fb28 service placement] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:20:22.732700 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf0e5c94-2167-4feb-997b-094b38f7fb28 service placement] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:20:22.737668 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58251516-cb7a-41d5-b2e5-b57a1ac7c89b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:20:22.738269 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58251516-cb7a-41d5-b2e5-b57a1ac7c89b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:20:22.738541 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58251516-cb7a-41d5-b2e5-b57a1ac7c89b admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:20:22.738807 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58251516-cb7a-41d5-b2e5-b57a1ac7c89b admin admin] PATH_INFO: `/v3/users/2087165506e54a42aaa5c41478405824` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:20:22.739922 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58251516-cb7a-41d5-b2e5-b57a1ac7c89b admin admin] RBAC: Authorizing `identity:delete_user(user_id=2087165506e54a42aaa5c41478405824)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:20:22.742075 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58251516-cb7a-41d5-b2e5-b57a1ac7c89b admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:20:22.790188 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-58251516-cb7a-41d5-b2e5-b57a1ac7c89b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2087165506e54a42aaa5c41478405824'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:20:22.798648 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-58251516-cb7a-41d5-b2e5-b57a1ac7c89b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2087165506e54a42aaa5c41478405824'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:20:22.811002 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-58251516-cb7a-41d5-b2e5-b57a1ac7c89b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2087165506e54a42aaa5c41478405824'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:20:22.817787 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-58251516-cb7a-41d5-b2e5-b57a1ac7c89b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2087165506e54a42aaa5c41478405824'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:20:22.818436 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-58251516-cb7a-41d5-b2e5-b57a1ac7c89b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2087165506e54a42aaa5c41478405824'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:20:22.824211 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1398/2765] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:20:22 2022] DELETE /identity/v3/users/2087165506e54a42aaa5c41478405824 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:20:22.828656 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e65f127-16a5-4c01-8338-15f5ab5f2390 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:20:22.828963 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e65f127-16a5-4c01-8338-15f5ab5f2390 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:20:22.829234 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e65f127-16a5-4c01-8338-15f5ab5f2390 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:20:22.848354 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-8e65f127-16a5-4c01-8338-15f5ab5f2390 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:20:22.849538 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-8e65f127-16a5-4c01-8338-15f5ab5f2390 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:20:22.908504 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1368/2766] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:20:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:20:22.918143 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58251516-cb7a-41d5-b2e5-b57a1ac7c89b admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:20:22.919562 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-58251516-cb7a-41d5-b2e5-b57a1ac7c89b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:20:22.999394 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58251516-cb7a-41d5-b2e5-b57a1ac7c89b admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:20:23.003560 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-784aaea4-78fa-463c-a823-b00b32591491 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:20:23.004154 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-784aaea4-78fa-463c-a823-b00b32591491 service neutron] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:20:23.004432 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-784aaea4-78fa-463c-a823-b00b32591491 service neutron] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:20:23.004702 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-784aaea4-78fa-463c-a823-b00b32591491 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:20:23.013605 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-784aaea4-78fa-463c-a823-b00b32591491 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:20:23.015336 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-784aaea4-78fa-463c-a823-b00b32591491 service neutron] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:20:23.024754 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1399/2767] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:20:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:20:23.291314 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e65f127-16a5-4c01-8338-15f5ab5f2390 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:20:23.292768 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-8e65f127-16a5-4c01-8338-15f5ab5f2390 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:20:23.305031 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e65f127-16a5-4c01-8338-15f5ab5f2390 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:20:23.309540 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eea649c5-9a97-4468-b0cf-0792fe71874b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:20:23.310160 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea649c5-9a97-4468-b0cf-0792fe71874b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:20:23.310555 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea649c5-9a97-4468-b0cf-0792fe71874b admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:20:23.310842 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea649c5-9a97-4468-b0cf-0792fe71874b admin admin] PATH_INFO: `/v3/projects/a2d6bd5a77fd46e696cfa95aa85def3a` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:20:23.311947 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eea649c5-9a97-4468-b0cf-0792fe71874b admin admin] RBAC: Authorizing `identity:delete_project(project_id=a2d6bd5a77fd46e696cfa95aa85def3a)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:20:23.313855 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eea649c5-9a97-4468-b0cf-0792fe71874b admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:20:23.365736 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-eea649c5-9a97-4468-b0cf-0792fe71874b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a2d6bd5a77fd46e696cfa95aa85def3a', 'request_id': 'req-eea649c5-9a97-4468-b0cf-0792fe71874b'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:20:23.374390 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-eea649c5-9a97-4468-b0cf-0792fe71874b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a2d6bd5a77fd46e696cfa95aa85def3a', 'request_id': 'req-eea649c5-9a97-4468-b0cf-0792fe71874b'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:20:23.381206 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-eea649c5-9a97-4468-b0cf-0792fe71874b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a2d6bd5a77fd46e696cfa95aa85def3a', 'request_id': 'req-eea649c5-9a97-4468-b0cf-0792fe71874b'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:20:23.392498 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-eea649c5-9a97-4468-b0cf-0792fe71874b admin admin] The token cache is being invalidate because project a2d6bd5a77fd46e696cfa95aa85def3a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81624) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:20:23.392834 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-eea649c5-9a97-4468-b0cf-0792fe71874b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:20:23.393507 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-eea649c5-9a97-4468-b0cf-0792fe71874b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:20:23.397753 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1369/2768] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:20:23 2022] DELETE /identity/v3/projects/a2d6bd5a77fd46e696cfa95aa85def3a => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 05:20:23.400186 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8c14832-623d-4dfb-b3fb-a05852160e77 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:20:23.400548 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8c14832-623d-4dfb-b3fb-a05852160e77 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:20:23.400897 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8c14832-623d-4dfb-b3fb-a05852160e77 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:20:23.409175 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:20:23.423342 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-f8c14832-623d-4dfb-b3fb-a05852160e77 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:20:23.424546 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-f8c14832-623d-4dfb-b3fb-a05852160e77 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:20:23.493715 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1400/2769] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:20:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:20:23.502180 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1b89bb-988d-41bf-94be-dd0cad30628a None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:20:23.502710 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1b89bb-988d-41bf-94be-dd0cad30628a None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:20:23.503117 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1b89bb-988d-41bf-94be-dd0cad30628a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:20:23.527559 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-3d1b89bb-988d-41bf-94be-dd0cad30628a None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:20:23.528747 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-3d1b89bb-988d-41bf-94be-dd0cad30628a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:20:23.537094 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1370/2770] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:20:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:20:23.541343 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8c14832-623d-4dfb-b3fb-a05852160e77 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:20:23.550975 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8c14832-623d-4dfb-b3fb-a05852160e77 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:20:23.555508 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42aa7fe0-c34f-481f-9547-70b9f9400fa2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:20:23.556093 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42aa7fe0-c34f-481f-9547-70b9f9400fa2 admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:20:23.556370 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42aa7fe0-c34f-481f-9547-70b9f9400fa2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:20:23.556639 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42aa7fe0-c34f-481f-9547-70b9f9400fa2 admin admin] PATH_INFO: `/v3/domains` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:20:23.557298 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42aa7fe0-c34f-481f-9547-70b9f9400fa2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:20:23.557596 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42aa7fe0-c34f-481f-9547-70b9f9400fa2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:20:23.559237 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42aa7fe0-c34f-481f-9547-70b9f9400fa2 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:20:23.571298 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1401/2771] 10.222.0.77 () {58 vars in 1153 bytes} [Tue Aug 30 05:20:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:20:23.575516 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d1b89bb-988d-41bf-94be-dd0cad30628a None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:20:23.585534 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d1b89bb-988d-41bf-94be-dd0cad30628a None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:20:23.590500 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-942dffd4-0bc5-49f1-9ae4-377f30fa4279 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:20:23.591127 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-942dffd4-0bc5-49f1-9ae4-377f30fa4279 admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:20:23.591405 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-942dffd4-0bc5-49f1-9ae4-377f30fa4279 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:20:23.591674 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-942dffd4-0bc5-49f1-9ae4-377f30fa4279 admin admin] PATH_INFO: `/v3/projects` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:20:23.592289 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-942dffd4-0bc5-49f1-9ae4-377f30fa4279 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:20:23.594209 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-942dffd4-0bc5-49f1-9ae4-377f30fa4279 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:20:23.620593 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1371/2772] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:20:23 2022] POST /identity/v3/projects => generated 399 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:20:23.624714 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42aa7fe0-c34f-481f-9547-70b9f9400fa2 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:20:23.635236 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42aa7fe0-c34f-481f-9547-70b9f9400fa2 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:20:23.693304 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8695c8c2-1357-4662-a28a-319ac0439578 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:20:23.693996 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8695c8c2-1357-4662-a28a-319ac0439578 admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:20:23.694173 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8695c8c2-1357-4662-a28a-319ac0439578 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:20:23.694444 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8695c8c2-1357-4662-a28a-319ac0439578 admin admin] PATH_INFO: `/v3/users` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:20:23.695101 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8695c8c2-1357-4662-a28a-319ac0439578 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:20:23.697016 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8695c8c2-1357-4662-a28a-319ac0439578 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:20:23.718695 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1402/2773] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:20:23 2022] POST /identity/v3/users => generated 300 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:20:23.723147 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-942dffd4-0bc5-49f1-9ae4-377f30fa4279 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:20:23.733262 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-942dffd4-0bc5-49f1-9ae4-377f30fa4279 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:20:23.737791 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd721909-d083-4069-a555-e0338a6b449e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:20:23.738376 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd721909-d083-4069-a555-e0338a6b449e admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:20:23.738651 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd721909-d083-4069-a555-e0338a6b449e admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:20:23.738975 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd721909-d083-4069-a555-e0338a6b449e admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:20:23.739611 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd721909-d083-4069-a555-e0338a6b449e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:20:23.741187 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd721909-d083-4069-a555-e0338a6b449e admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:20:23.750934 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1372/2774] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:20:23 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:20:23.755258 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8695c8c2-1357-4662-a28a-319ac0439578 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:20:23.765813 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8695c8c2-1357-4662-a28a-319ac0439578 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:20:23.771092 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f650f583-3f0f-471a-a5c6-b69aab4b71dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:20:23.771716 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f650f583-3f0f-471a-a5c6-b69aab4b71dc admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:20:23.772008 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f650f583-3f0f-471a-a5c6-b69aab4b71dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:20:23.772281 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f650f583-3f0f-471a-a5c6-b69aab4b71dc admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:20:23.772915 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f650f583-3f0f-471a-a5c6-b69aab4b71dc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:20:23.774504 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f650f583-3f0f-471a-a5c6-b69aab4b71dc admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:20:23.784883 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1403/2775] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:20:23 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:20:23.788941 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd721909-d083-4069-a555-e0338a6b449e admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:20:23.798845 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd721909-d083-4069-a555-e0338a6b449e admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:20:23.803693 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f22eaac-98e0-4451-9e22-6a5552c907a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:20:23.804424 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f22eaac-98e0-4451-9e22-6a5552c907a2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:20:23.804707 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f22eaac-98e0-4451-9e22-6a5552c907a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:20:23.804976 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f22eaac-98e0-4451-9e22-6a5552c907a2 admin admin] PATH_INFO: `/v3/projects/2422a57ea9e64daa9d3fa39792b17f9c/users/144b983fd6fe425d870fd3b84b832401/roles/6f03d86980c34c36b44a8d6764ff4fda` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:20:23.826696 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f22eaac-98e0-4451-9e22-6a5552c907a2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2422a57ea9e64daa9d3fa39792b17f9c, user_id=144b983fd6fe425d870fd3b84b832401, role_id=6f03d86980c34c36b44a8d6764ff4fda)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:20:23.829954 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f22eaac-98e0-4451-9e22-6a5552c907a2 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:20:23.854114 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1373/2776] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:20:23 2022] PUT /identity/v3/projects/2422a57ea9e64daa9d3fa39792b17f9c/users/144b983fd6fe425d870fd3b84b832401/roles/6f03d86980c34c36b44a8d6764ff4fda => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:20:23.855494 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94b0723e-b0be-4cc9-bf60-1cbe3722bd26 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:20:23.855779 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94b0723e-b0be-4cc9-bf60-1cbe3722bd26 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:20:23.856043 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94b0723e-b0be-4cc9-bf60-1cbe3722bd26 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:20:23.885093 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-94b0723e-b0be-4cc9-bf60-1cbe3722bd26 None None] MFA Rules not processed for user `144b983fd6fe425d870fd3b84b832401`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:20:23.886304 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-94b0723e-b0be-4cc9-bf60-1cbe3722bd26 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:20:23.945406 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:20:23.951134 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1404/2777] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:20:23 2022] POST /identity/v3/auth/tokens => generated 3276 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:20:23.962044 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f22eaac-98e0-4451-9e22-6a5552c907a2 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:20:23.963718 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-7f22eaac-98e0-4451-9e22-6a5552c907a2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:20:23.986883 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:20:24.043124 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f22eaac-98e0-4451-9e22-6a5552c907a2 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:20:24.046805 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f20094c4-f249-4914-ab7c-aa5486a7f3f3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '019a79b2affe478b8a30d1c538d0415c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:20:24.047432 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f20094c4-f249-4914-ab7c-aa5486a7f3f3 service cinder] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:20:24.047716 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f20094c4-f249-4914-ab7c-aa5486a7f3f3 service cinder] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:20:24.048492 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f20094c4-f249-4914-ab7c-aa5486a7f3f3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:20:24.056788 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f20094c4-f249-4914-ab7c-aa5486a7f3f3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:20:24.058206 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f20094c4-f249-4914-ab7c-aa5486a7f3f3 service cinder] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:20:24.067048 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1374/2778] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:20:23 2022] GET /identity/v3/auth/tokens => generated 3276 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:20:30.476317 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94b0723e-b0be-4cc9-bf60-1cbe3722bd26 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:20:30.477912 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-94b0723e-b0be-4cc9-bf60-1cbe3722bd26 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:20:30.534800 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94b0723e-b0be-4cc9-bf60-1cbe3722bd26 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:20:30.538359 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da61125f-76d3-4b3d-9cb9-7f0fe1aca04f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:20:30.538986 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da61125f-76d3-4b3d-9cb9-7f0fe1aca04f service neutron] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:20:30.539259 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da61125f-76d3-4b3d-9cb9-7f0fe1aca04f service neutron] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:20:30.539521 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da61125f-76d3-4b3d-9cb9-7f0fe1aca04f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:20:30.541202 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-da61125f-76d3-4b3d-9cb9-7f0fe1aca04f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:20:30.601598 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da61125f-76d3-4b3d-9cb9-7f0fe1aca04f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:20:30.603043 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da61125f-76d3-4b3d-9cb9-7f0fe1aca04f service neutron] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:20:30.613045 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1405/2779] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:20:30 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 137 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:20:32.129203 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f20094c4-f249-4914-ab7c-aa5486a7f3f3 service cinder] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:20:32.207889 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f20094c4-f249-4914-ab7c-aa5486a7f3f3 service cinder] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:20:32.212874 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2145012-d591-4446-9f15-d422b15bedd3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:20:32.213658 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2145012-d591-4446-9f15-d422b15bedd3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:20:32.214048 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2145012-d591-4446-9f15-d422b15bedd3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:20:32.214258 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2145012-d591-4446-9f15-d422b15bedd3 admin admin] PATH_INFO: `/v3/users/144b983fd6fe425d870fd3b84b832401` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:20:32.216104 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2145012-d591-4446-9f15-d422b15bedd3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=144b983fd6fe425d870fd3b84b832401)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:20:32.218037 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2145012-d591-4446-9f15-d422b15bedd3 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:20:32.274538 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-b2145012-d591-4446-9f15-d422b15bedd3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '144b983fd6fe425d870fd3b84b832401'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:20:32.275261 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-b2145012-d591-4446-9f15-d422b15bedd3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '144b983fd6fe425d870fd3b84b832401'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:20:32.276315 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-b2145012-d591-4446-9f15-d422b15bedd3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '144b983fd6fe425d870fd3b84b832401'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:20:32.295127 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-b2145012-d591-4446-9f15-d422b15bedd3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '144b983fd6fe425d870fd3b84b832401'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:20:32.306183 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-b2145012-d591-4446-9f15-d422b15bedd3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '144b983fd6fe425d870fd3b84b832401'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:20:32.327450 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1375/2780] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:20:32 2022] DELETE /identity/v3/users/144b983fd6fe425d870fd3b84b832401 => generated 0 bytes in 197 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:20:32.330269 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d283eae3-9312-40ce-bc40-38d75c72a7ae None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:20:32.330727 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d283eae3-9312-40ce-bc40-38d75c72a7ae None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:20:32.330810 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d283eae3-9312-40ce-bc40-38d75c72a7ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:20:32.351687 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-d283eae3-9312-40ce-bc40-38d75c72a7ae None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:20:32.353091 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-d283eae3-9312-40ce-bc40-38d75c72a7ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:20:32.418238 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1406/2781] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:20:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:20:32.431224 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2145012-d591-4446-9f15-d422b15bedd3 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:20:32.432537 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-b2145012-d591-4446-9f15-d422b15bedd3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:20:32.487583 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:20:32.541575 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2145012-d591-4446-9f15-d422b15bedd3 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:20:32.547504 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b5cdb5c-64cd-4155-af8d-d677874535b3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:20:32.549486 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b5cdb5c-64cd-4155-af8d-d677874535b3 service neutron] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:20:32.549979 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b5cdb5c-64cd-4155-af8d-d677874535b3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:20:32.550324 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b5cdb5c-64cd-4155-af8d-d677874535b3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:20:32.564954 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b5cdb5c-64cd-4155-af8d-d677874535b3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:20:32.567204 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b5cdb5c-64cd-4155-af8d-d677874535b3 service neutron] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:20:32.580981 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1376/2782] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:20:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 150 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:20:33.087312 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d283eae3-9312-40ce-bc40-38d75c72a7ae None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:20:33.088699 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-d283eae3-9312-40ce-bc40-38d75c72a7ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:20:33.101270 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d283eae3-9312-40ce-bc40-38d75c72a7ae None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:20:33.106160 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad3354ab-4ea4-43ab-8729-3a6b39109302 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:20:33.106788 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad3354ab-4ea4-43ab-8729-3a6b39109302 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:20:33.107103 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad3354ab-4ea4-43ab-8729-3a6b39109302 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:20:33.107372 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad3354ab-4ea4-43ab-8729-3a6b39109302 admin admin] PATH_INFO: `/v3/projects/2422a57ea9e64daa9d3fa39792b17f9c` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:20:33.108531 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad3354ab-4ea4-43ab-8729-3a6b39109302 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2422a57ea9e64daa9d3fa39792b17f9c)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:20:33.110469 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad3354ab-4ea4-43ab-8729-3a6b39109302 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:20:33.162470 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:20:33.168788 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-ad3354ab-4ea4-43ab-8729-3a6b39109302 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2422a57ea9e64daa9d3fa39792b17f9c', 'request_id': 'req-ad3354ab-4ea4-43ab-8729-3a6b39109302'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:20:33.180483 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-ad3354ab-4ea4-43ab-8729-3a6b39109302 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2422a57ea9e64daa9d3fa39792b17f9c', 'request_id': 'req-ad3354ab-4ea4-43ab-8729-3a6b39109302'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:20:33.187332 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-ad3354ab-4ea4-43ab-8729-3a6b39109302 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2422a57ea9e64daa9d3fa39792b17f9c', 'request_id': 'req-ad3354ab-4ea4-43ab-8729-3a6b39109302'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:20:33.199342 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-ad3354ab-4ea4-43ab-8729-3a6b39109302 admin admin] The token cache is being invalidate because project 2422a57ea9e64daa9d3fa39792b17f9c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81625) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:20:33.199634 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-ad3354ab-4ea4-43ab-8729-3a6b39109302 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:20:33.200287 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-ad3354ab-4ea4-43ab-8729-3a6b39109302 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:20:33.202041 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1407/2783] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:20:33 2022] DELETE /identity/v3/projects/2422a57ea9e64daa9d3fa39792b17f9c => generated 0 bytes in 115 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:20:37.012117 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b5cdb5c-64cd-4155-af8d-d677874535b3 service neutron] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:20:37.013500 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-6b5cdb5c-64cd-4155-af8d-d677874535b3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:20:37.025101 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b5cdb5c-64cd-4155-af8d-d677874535b3 service neutron] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:20:37.073053 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1659602a-b7c4-4bee-9856-494ecc79277b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:20:37.073640 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1659602a-b7c4-4bee-9856-494ecc79277b service neutron] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:20:37.073915 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1659602a-b7c4-4bee-9856-494ecc79277b service neutron] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:20:37.074176 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1659602a-b7c4-4bee-9856-494ecc79277b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:20:37.075897 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-1659602a-b7c4-4bee-9856-494ecc79277b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:20:37.137383 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1659602a-b7c4-4bee-9856-494ecc79277b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:20:37.138804 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1659602a-b7c4-4bee-9856-494ecc79277b service neutron] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:20:37.148839 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1377/2784] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:20:37 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 137 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 05:20:43.195989 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=8,keystone:DELETE=4,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:20:45.135131 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e62e7e2-faaa-4bbe-b999-f7f2c6dfb046 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:20:45.135747 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e62e7e2-faaa-4bbe-b999-f7f2c6dfb046 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:20:45.135747 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e62e7e2-faaa-4bbe-b999-f7f2c6dfb046 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:20:45.154545 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-4e62e7e2-faaa-4bbe-b999-f7f2c6dfb046 None None] MFA Rules not processed for user `d079e0264f0e40db9fc93b8f217a7a74`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:20:45.155786 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-4e62e7e2-faaa-4bbe-b999-f7f2c6dfb046 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:20:45.208323 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1408/2785] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:20:45 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 74 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:20:47.144680 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=66 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:20:55.201362 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=25 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:21:51.106965 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-308d48e9-5954-45e2-a27e-3eab1769f422 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:21:51.107488 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-308d48e9-5954-45e2-a27e-3eab1769f422 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:21:51.107589 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-308d48e9-5954-45e2-a27e-3eab1769f422 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:21:51.126654 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-308d48e9-5954-45e2-a27e-3eab1769f422 None None] MFA Rules not processed for user `d079e0264f0e40db9fc93b8f217a7a74`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:21:51.128328 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-308d48e9-5954-45e2-a27e-3eab1769f422 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:21:51.137020 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1378/2786] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:21:51 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:21:53.780232 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b40bb14-b52c-4a7b-985f-04a86107146e None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:21:53.780718 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b40bb14-b52c-4a7b-985f-04a86107146e None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:21:53.780821 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b40bb14-b52c-4a7b-985f-04a86107146e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:21:53.812639 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-3b40bb14-b52c-4a7b-985f-04a86107146e None None] MFA Rules not processed for user `fe51dbec91624ed1ab00b441e29a3dd5`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:21:53.813854 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-3b40bb14-b52c-4a7b-985f-04a86107146e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:21:53.876547 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1409/2787] 10.222.0.77 () {58 vars in 967 bytes} [Tue Aug 30 05:21:53 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 05:21:53.923447 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-308d48e9-5954-45e2-a27e-3eab1769f422 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:21:53.925697 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-308d48e9-5954-45e2-a27e-3eab1769f422 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:21:54.005706 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-308d48e9-5954-45e2-a27e-3eab1769f422 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:21:54.012900 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77f75f4f-633e-499e-9cb8-32bd91eddeb7 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a7f7afc3d44ec788c05d005e52a1b8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:21:54.013409 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f75f4f-633e-499e-9cb8-32bd91eddeb7 service swift] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:21:54.013686 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f75f4f-633e-499e-9cb8-32bd91eddeb7 service swift] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:21:54.013946 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f75f4f-633e-499e-9cb8-32bd91eddeb7 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:21:54.025097 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77f75f4f-633e-499e-9cb8-32bd91eddeb7 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:21:54.026716 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77f75f4f-633e-499e-9cb8-32bd91eddeb7 service swift] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:21:54.035871 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1379/2788] 10.222.0.77 () {60 vars in 1384 bytes} [Tue Aug 30 05:21:53 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 113 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Aug 30 05:22:01.259376 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b40bb14-b52c-4a7b-985f-04a86107146e None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:22:01.260837 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-3b40bb14-b52c-4a7b-985f-04a86107146e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:22:01.326915 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b40bb14-b52c-4a7b-985f-04a86107146e None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:22:01.331955 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eaf7f2b-467c-4987-90fd-54d96cae08bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:22:01.332571 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eaf7f2b-467c-4987-90fd-54d96cae08bd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:22:01.332851 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eaf7f2b-467c-4987-90fd-54d96cae08bd admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:22:01.333122 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eaf7f2b-467c-4987-90fd-54d96cae08bd admin admin] PATH_INFO: `/v3/users/b92e3e3eff1e40d38fdfc68e467367f7` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:22:01.334240 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eaf7f2b-467c-4987-90fd-54d96cae08bd admin admin] RBAC: Authorizing `identity:delete_user(user_id=b92e3e3eff1e40d38fdfc68e467367f7)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:22:01.336181 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eaf7f2b-467c-4987-90fd-54d96cae08bd admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:22:01.382162 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-8eaf7f2b-467c-4987-90fd-54d96cae08bd admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b92e3e3eff1e40d38fdfc68e467367f7'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:22:01.388985 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-8eaf7f2b-467c-4987-90fd-54d96cae08bd admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b92e3e3eff1e40d38fdfc68e467367f7'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:22:01.401255 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-8eaf7f2b-467c-4987-90fd-54d96cae08bd admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b92e3e3eff1e40d38fdfc68e467367f7'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:22:01.408318 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-8eaf7f2b-467c-4987-90fd-54d96cae08bd admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b92e3e3eff1e40d38fdfc68e467367f7'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:22:01.409033 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-8eaf7f2b-467c-4987-90fd-54d96cae08bd admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b92e3e3eff1e40d38fdfc68e467367f7'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:22:01.414756 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1410/2789] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:22:01 2022] DELETE /identity/v3/users/b92e3e3eff1e40d38fdfc68e467367f7 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:22:01.587972 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77f75f4f-633e-499e-9cb8-32bd91eddeb7 service swift] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:22:01.600269 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77f75f4f-633e-499e-9cb8-32bd91eddeb7 service swift] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:22:01.785532 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3da2aa70-a673-4f97-887d-4e8f65668258 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:22:01.786133 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da2aa70-a673-4f97-887d-4e8f65668258 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:22:01.786405 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da2aa70-a673-4f97-887d-4e8f65668258 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:22:01.786675 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da2aa70-a673-4f97-887d-4e8f65668258 admin admin] PATH_INFO: `/v3/projects/4826d4507ffc4e3288aac00b1409ea46` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:22:01.787790 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3da2aa70-a673-4f97-887d-4e8f65668258 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4826d4507ffc4e3288aac00b1409ea46)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:22:01.789659 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3da2aa70-a673-4f97-887d-4e8f65668258 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:22:01.834985 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-3da2aa70-a673-4f97-887d-4e8f65668258 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4826d4507ffc4e3288aac00b1409ea46', 'request_id': 'req-3da2aa70-a673-4f97-887d-4e8f65668258'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:22:01.842183 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-3da2aa70-a673-4f97-887d-4e8f65668258 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4826d4507ffc4e3288aac00b1409ea46', 'request_id': 'req-3da2aa70-a673-4f97-887d-4e8f65668258'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:22:01.848126 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-3da2aa70-a673-4f97-887d-4e8f65668258 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4826d4507ffc4e3288aac00b1409ea46', 'request_id': 'req-3da2aa70-a673-4f97-887d-4e8f65668258'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:22:01.858025 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-3da2aa70-a673-4f97-887d-4e8f65668258 admin admin] The token cache is being invalidate because project 4826d4507ffc4e3288aac00b1409ea46 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81624) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:22:01.858306 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-3da2aa70-a673-4f97-887d-4e8f65668258 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:22:01.858944 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-3da2aa70-a673-4f97-887d-4e8f65668258 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:22:01.860374 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1380/2790] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:22:01 2022] DELETE /identity/v3/projects/4826d4507ffc4e3288aac00b1409ea46 => generated 0 bytes in 273 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 05:22:01.864873 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eaf7f2b-467c-4987-90fd-54d96cae08bd admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:22:01.866278 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-8eaf7f2b-467c-4987-90fd-54d96cae08bd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:22:01.879729 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eaf7f2b-467c-4987-90fd-54d96cae08bd admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:22:01.884221 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:22:01.936487 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbc3d6b5-f9d3-4786-8c42-6146ab667e8a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:22:01.937052 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbc3d6b5-f9d3-4786-8c42-6146ab667e8a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:22:01.937331 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbc3d6b5-f9d3-4786-8c42-6146ab667e8a admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:22:01.937604 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbc3d6b5-f9d3-4786-8c42-6146ab667e8a admin admin] PATH_INFO: `/v3/users/8cc19d2d467a40cfa9cd51c3e4861b29` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:22:01.938774 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbc3d6b5-f9d3-4786-8c42-6146ab667e8a admin admin] RBAC: Authorizing `identity:delete_user(user_id=8cc19d2d467a40cfa9cd51c3e4861b29)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:22:01.941342 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbc3d6b5-f9d3-4786-8c42-6146ab667e8a admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:22:01.985865 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-cbc3d6b5-f9d3-4786-8c42-6146ab667e8a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8cc19d2d467a40cfa9cd51c3e4861b29'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:22:01.993306 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-cbc3d6b5-f9d3-4786-8c42-6146ab667e8a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8cc19d2d467a40cfa9cd51c3e4861b29'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:22:02.005684 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-cbc3d6b5-f9d3-4786-8c42-6146ab667e8a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8cc19d2d467a40cfa9cd51c3e4861b29'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:22:02.012710 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-cbc3d6b5-f9d3-4786-8c42-6146ab667e8a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8cc19d2d467a40cfa9cd51c3e4861b29'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:22:02.013384 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-cbc3d6b5-f9d3-4786-8c42-6146ab667e8a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8cc19d2d467a40cfa9cd51c3e4861b29'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:22:02.019322 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1411/2791] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:22:01 2022] DELETE /identity/v3/users/8cc19d2d467a40cfa9cd51c3e4861b29 => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:22:02.199990 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3da2aa70-a673-4f97-887d-4e8f65668258 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:22:02.202299 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-3da2aa70-a673-4f97-887d-4e8f65668258 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:22:02.216087 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3da2aa70-a673-4f97-887d-4e8f65668258 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:22:02.226788 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:22:02.277857 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad098bc9-1cb3-4a9c-a59d-647240b665fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:22:02.278481 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad098bc9-1cb3-4a9c-a59d-647240b665fa admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:22:02.278755 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad098bc9-1cb3-4a9c-a59d-647240b665fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:22:02.279051 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad098bc9-1cb3-4a9c-a59d-647240b665fa admin admin] PATH_INFO: `/v3/projects/b530b1e42d404763b5da765efae99119` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:22:02.280169 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad098bc9-1cb3-4a9c-a59d-647240b665fa admin admin] RBAC: Authorizing `identity:delete_project(project_id=b530b1e42d404763b5da765efae99119)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:22:02.282113 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad098bc9-1cb3-4a9c-a59d-647240b665fa admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:22:02.336096 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-ad098bc9-1cb3-4a9c-a59d-647240b665fa admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b530b1e42d404763b5da765efae99119', 'request_id': 'req-ad098bc9-1cb3-4a9c-a59d-647240b665fa'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:22:02.343391 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-ad098bc9-1cb3-4a9c-a59d-647240b665fa admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b530b1e42d404763b5da765efae99119', 'request_id': 'req-ad098bc9-1cb3-4a9c-a59d-647240b665fa'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:22:02.350311 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-ad098bc9-1cb3-4a9c-a59d-647240b665fa admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b530b1e42d404763b5da765efae99119', 'request_id': 'req-ad098bc9-1cb3-4a9c-a59d-647240b665fa'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:22:02.361178 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-ad098bc9-1cb3-4a9c-a59d-647240b665fa admin admin] The token cache is being invalidate because project b530b1e42d404763b5da765efae99119 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81624) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:22:02.361469 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-ad098bc9-1cb3-4a9c-a59d-647240b665fa admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:22:02.362730 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-ad098bc9-1cb3-4a9c-a59d-647240b665fa admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:22:02.364375 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1381/2792] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:22:02 2022] DELETE /identity/v3/projects/b530b1e42d404763b5da765efae99119 => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:22:12.015783 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=49,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:22:12.358304 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=45,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:22:26.611093 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbc3d6b5-f9d3-4786-8c42-6146ab667e8a admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:22:26.612571 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-cbc3d6b5-f9d3-4786-8c42-6146ab667e8a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:22:26.671464 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbc3d6b5-f9d3-4786-8c42-6146ab667e8a admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:22:26.720943 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb641670-429e-428f-a9ae-1f1f18bb5e35 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:22:26.722147 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb641670-429e-428f-a9ae-1f1f18bb5e35 service neutron] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:22:26.722404 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb641670-429e-428f-a9ae-1f1f18bb5e35 service neutron] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:22:26.722670 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb641670-429e-428f-a9ae-1f1f18bb5e35 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:22:26.724497 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-bb641670-429e-428f-a9ae-1f1f18bb5e35 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:22:26.808833 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb641670-429e-428f-a9ae-1f1f18bb5e35 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:22:26.810295 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb641670-429e-428f-a9ae-1f1f18bb5e35 service neutron] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:22:26.833117 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1412/2793] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:22:26 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 223 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:22:28.018851 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad098bc9-1cb3-4a9c-a59d-647240b665fa admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:22:28.020409 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-ad098bc9-1cb3-4a9c-a59d-647240b665fa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:22:28.033053 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad098bc9-1cb3-4a9c-a59d-647240b665fa admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:22:28.037350 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56ba6cbe-3497-4c41-af05-f8890a016122 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:22:28.038040 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ba6cbe-3497-4c41-af05-f8890a016122 service neutron] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:22:28.038381 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ba6cbe-3497-4c41-af05-f8890a016122 service neutron] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:22:28.038713 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ba6cbe-3497-4c41-af05-f8890a016122 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:22:28.040533 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-56ba6cbe-3497-4c41-af05-f8890a016122 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:22:28.054854 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56ba6cbe-3497-4c41-af05-f8890a016122 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:22:28.056288 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56ba6cbe-3497-4c41-af05-f8890a016122 service neutron] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:22:28.067038 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1382/2794] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:22:28 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 49 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:22:32.210109 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb641670-429e-428f-a9ae-1f1f18bb5e35 service neutron] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:22:32.211543 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-bb641670-429e-428f-a9ae-1f1f18bb5e35 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:22:32.223020 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb641670-429e-428f-a9ae-1f1f18bb5e35 service neutron] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:22:32.226755 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c23f721-902a-4c2f-b2f2-24b6fa3a3b43 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:22:32.227383 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c23f721-902a-4c2f-b2f2-24b6fa3a3b43 service neutron] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:22:32.227665 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c23f721-902a-4c2f-b2f2-24b6fa3a3b43 service neutron] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:22:32.227944 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c23f721-902a-4c2f-b2f2-24b6fa3a3b43 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:22:32.229974 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-1c23f721-902a-4c2f-b2f2-24b6fa3a3b43 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:22:32.267473 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:22:32.296299 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c23f721-902a-4c2f-b2f2-24b6fa3a3b43 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:22:32.297831 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c23f721-902a-4c2f-b2f2-24b6fa3a3b43 service neutron] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:22:32.307946 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1413/2795] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:22:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:22:35.302773 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56ba6cbe-3497-4c41-af05-f8890a016122 service neutron] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:22:35.304224 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-56ba6cbe-3497-4c41-af05-f8890a016122 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:22:35.400684 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56ba6cbe-3497-4c41-af05-f8890a016122 service neutron] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:22:35.405960 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-329a9fae-ff95-4933-9f8e-e8854b90d182 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1972959402704b89925001cd8d90cb0e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:22:35.406553 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-329a9fae-ff95-4933-9f8e-e8854b90d182 service placement] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:22:35.406848 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-329a9fae-ff95-4933-9f8e-e8854b90d182 service placement] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:22:35.407119 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-329a9fae-ff95-4933-9f8e-e8854b90d182 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:22:35.410388 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-329a9fae-ff95-4933-9f8e-e8854b90d182 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:22:35.422582 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-329a9fae-ff95-4933-9f8e-e8854b90d182 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:22:35.424085 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-329a9fae-ff95-4933-9f8e-e8854b90d182 service placement] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:22:35.441277 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1383/2796] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:22:35 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 137 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:22:38.421538 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c23f721-902a-4c2f-b2f2-24b6fa3a3b43 service neutron] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:22:38.423049 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-1c23f721-902a-4c2f-b2f2-24b6fa3a3b43 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:22:38.437568 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c23f721-902a-4c2f-b2f2-24b6fa3a3b43 service neutron] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:22:38.443756 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a430889-9e6f-4686-b74f-44ed69b592c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:22:38.444383 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a430889-9e6f-4686-b74f-44ed69b592c2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:22:38.444680 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a430889-9e6f-4686-b74f-44ed69b592c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:22:38.444932 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a430889-9e6f-4686-b74f-44ed69b592c2 admin admin] PATH_INFO: `/v3/users/9256fd43e5124fb68d4e5453584d1e8d` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:22:38.446044 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a430889-9e6f-4686-b74f-44ed69b592c2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9256fd43e5124fb68d4e5453584d1e8d)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:22:38.448030 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a430889-9e6f-4686-b74f-44ed69b592c2 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:22:38.494324 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-2a430889-9e6f-4686-b74f-44ed69b592c2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9256fd43e5124fb68d4e5453584d1e8d'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:22:38.502887 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-2a430889-9e6f-4686-b74f-44ed69b592c2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9256fd43e5124fb68d4e5453584d1e8d'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:22:38.514765 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-2a430889-9e6f-4686-b74f-44ed69b592c2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9256fd43e5124fb68d4e5453584d1e8d'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:22:38.521973 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-2a430889-9e6f-4686-b74f-44ed69b592c2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9256fd43e5124fb68d4e5453584d1e8d'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:22:38.522609 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-2a430889-9e6f-4686-b74f-44ed69b592c2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9256fd43e5124fb68d4e5453584d1e8d'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:22:38.528338 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1414/2797] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:22:38 2022] DELETE /identity/v3/users/9256fd43e5124fb68d4e5453584d1e8d => generated 0 bytes in 107 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:22:38.696311 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-329a9fae-ff95-4933-9f8e-e8854b90d182 service placement] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:22:38.707609 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-329a9fae-ff95-4933-9f8e-e8854b90d182 service placement] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:22:38.762669 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60d84182-9e62-4677-b800-9235ca1b78c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:22:38.763312 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60d84182-9e62-4677-b800-9235ca1b78c4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:22:38.763584 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60d84182-9e62-4677-b800-9235ca1b78c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:22:38.763850 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60d84182-9e62-4677-b800-9235ca1b78c4 admin admin] PATH_INFO: `/v3/projects/97e3452b02294219ae56cd23eb2aa0ce` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:22:38.764931 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60d84182-9e62-4677-b800-9235ca1b78c4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=97e3452b02294219ae56cd23eb2aa0ce)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:22:38.766795 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60d84182-9e62-4677-b800-9235ca1b78c4 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:22:38.810987 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-60d84182-9e62-4677-b800-9235ca1b78c4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '97e3452b02294219ae56cd23eb2aa0ce', 'request_id': 'req-60d84182-9e62-4677-b800-9235ca1b78c4'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:22:38.818251 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-60d84182-9e62-4677-b800-9235ca1b78c4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '97e3452b02294219ae56cd23eb2aa0ce', 'request_id': 'req-60d84182-9e62-4677-b800-9235ca1b78c4'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:22:38.824356 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-60d84182-9e62-4677-b800-9235ca1b78c4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '97e3452b02294219ae56cd23eb2aa0ce', 'request_id': 'req-60d84182-9e62-4677-b800-9235ca1b78c4'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:22:38.834878 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-60d84182-9e62-4677-b800-9235ca1b78c4 admin admin] The token cache is being invalidate because project 97e3452b02294219ae56cd23eb2aa0ce was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81624) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:22:38.835162 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-60d84182-9e62-4677-b800-9235ca1b78c4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:22:38.835772 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-60d84182-9e62-4677-b800-9235ca1b78c4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:22:38.837262 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1384/2798] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:22:38 2022] DELETE /identity/v3/projects/97e3452b02294219ae56cd23eb2aa0ce => generated 0 bytes in 141 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:22:38.841494 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a430889-9e6f-4686-b74f-44ed69b592c2 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:22:38.842937 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-2a430889-9e6f-4686-b74f-44ed69b592c2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:22:38.855674 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a430889-9e6f-4686-b74f-44ed69b592c2 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:22:38.910935 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30952d72-9cc9-4383-914d-38840698abaf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:22:38.911485 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30952d72-9cc9-4383-914d-38840698abaf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:22:38.911759 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30952d72-9cc9-4383-914d-38840698abaf admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:22:38.912030 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30952d72-9cc9-4383-914d-38840698abaf admin admin] PATH_INFO: `/v3/users/910c0e6f6a8646cf8671f1efc96424e1` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:22:38.913143 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30952d72-9cc9-4383-914d-38840698abaf admin admin] RBAC: Authorizing `identity:delete_user(user_id=910c0e6f6a8646cf8671f1efc96424e1)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:22:38.915067 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30952d72-9cc9-4383-914d-38840698abaf admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:22:38.961610 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=82,keystone:DELETE=17,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:22:38.963202 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-30952d72-9cc9-4383-914d-38840698abaf admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '910c0e6f6a8646cf8671f1efc96424e1'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:22:38.972945 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-30952d72-9cc9-4383-914d-38840698abaf admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '910c0e6f6a8646cf8671f1efc96424e1'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:22:38.985312 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-30952d72-9cc9-4383-914d-38840698abaf admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '910c0e6f6a8646cf8671f1efc96424e1'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:22:38.992123 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-30952d72-9cc9-4383-914d-38840698abaf admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '910c0e6f6a8646cf8671f1efc96424e1'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:22:38.992752 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-30952d72-9cc9-4383-914d-38840698abaf admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '910c0e6f6a8646cf8671f1efc96424e1'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:22:38.998507 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1415/2799] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:22:38 2022] DELETE /identity/v3/users/910c0e6f6a8646cf8671f1efc96424e1 => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:22:39.167278 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60d84182-9e62-4677-b800-9235ca1b78c4 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:22:39.169149 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-60d84182-9e62-4677-b800-9235ca1b78c4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:22:39.186339 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60d84182-9e62-4677-b800-9235ca1b78c4 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:22:39.190544 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:22:39.246915 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-554b4511-7571-4647-828c-3d34ffa8a680 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:22:39.247538 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-554b4511-7571-4647-828c-3d34ffa8a680 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:22:39.247814 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-554b4511-7571-4647-828c-3d34ffa8a680 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:22:39.248084 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-554b4511-7571-4647-828c-3d34ffa8a680 admin admin] PATH_INFO: `/v3/projects/3e3469ca8e2641179d0fc7988ed0e5c9` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:22:39.249182 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-554b4511-7571-4647-828c-3d34ffa8a680 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3e3469ca8e2641179d0fc7988ed0e5c9)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:22:39.251107 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-554b4511-7571-4647-828c-3d34ffa8a680 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:22:39.296140 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-554b4511-7571-4647-828c-3d34ffa8a680 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3e3469ca8e2641179d0fc7988ed0e5c9', 'request_id': 'req-554b4511-7571-4647-828c-3d34ffa8a680'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:22:39.303829 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-554b4511-7571-4647-828c-3d34ffa8a680 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3e3469ca8e2641179d0fc7988ed0e5c9', 'request_id': 'req-554b4511-7571-4647-828c-3d34ffa8a680'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:22:39.309815 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-554b4511-7571-4647-828c-3d34ffa8a680 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3e3469ca8e2641179d0fc7988ed0e5c9', 'request_id': 'req-554b4511-7571-4647-828c-3d34ffa8a680'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:22:39.320181 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-554b4511-7571-4647-828c-3d34ffa8a680 admin admin] The token cache is being invalidate because project 3e3469ca8e2641179d0fc7988ed0e5c9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81624) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:22:39.320467 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-554b4511-7571-4647-828c-3d34ffa8a680 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:22:39.321080 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-554b4511-7571-4647-828c-3d34ffa8a680 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:22:39.322488 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1385/2800] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:22:39 2022] DELETE /identity/v3/projects/3e3469ca8e2641179d0fc7988ed0e5c9 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:22:48.995069 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=13,keystone:INSERT=1,keystone:DELETE=3 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:22:49.317310 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=48,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:22:51.423270 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30952d72-9cc9-4383-914d-38840698abaf admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:22:51.424699 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-30952d72-9cc9-4383-914d-38840698abaf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:22:51.481742 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30952d72-9cc9-4383-914d-38840698abaf admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:22:51.529195 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6fcab93-501b-4260-bd3e-c17fea50a6dc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:22:51.529790 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6fcab93-501b-4260-bd3e-c17fea50a6dc service neutron] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:22:51.530064 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6fcab93-501b-4260-bd3e-c17fea50a6dc service neutron] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:22:51.530330 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6fcab93-501b-4260-bd3e-c17fea50a6dc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:22:51.532054 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-f6fcab93-501b-4260-bd3e-c17fea50a6dc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:22:51.589190 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6fcab93-501b-4260-bd3e-c17fea50a6dc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:22:51.590600 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6fcab93-501b-4260-bd3e-c17fea50a6dc service neutron] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:22:51.599284 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1416/2801] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:22:51 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 177 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:22:52.489863 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-554b4511-7571-4647-828c-3d34ffa8a680 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:22:52.491446 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-554b4511-7571-4647-828c-3d34ffa8a680 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:22:52.586865 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-554b4511-7571-4647-828c-3d34ffa8a680 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:22:52.595901 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8491be13-45d8-4bef-86d1-72af00cc46c4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '886eb5e00bbf4ec292eb962b665b37d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:22:52.597154 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8491be13-45d8-4bef-86d1-72af00cc46c4 service nova] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:22:52.598008 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8491be13-45d8-4bef-86d1-72af00cc46c4 service nova] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:22:52.598280 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8491be13-45d8-4bef-86d1-72af00cc46c4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:22:52.600658 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-8491be13-45d8-4bef-86d1-72af00cc46c4 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:22:52.616192 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8491be13-45d8-4bef-86d1-72af00cc46c4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:22:52.617816 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8491be13-45d8-4bef-86d1-72af00cc46c4 service nova] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:22:52.643994 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1386/2802] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:22:52 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 155 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:22:57.584479 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6fcab93-501b-4260-bd3e-c17fea50a6dc service neutron] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:22:57.585887 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-f6fcab93-501b-4260-bd3e-c17fea50a6dc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:22:57.596688 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6fcab93-501b-4260-bd3e-c17fea50a6dc service neutron] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:22:57.600216 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25f3d775-260b-4320-ba12-ccc076c1524a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:22:57.600794 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25f3d775-260b-4320-ba12-ccc076c1524a service neutron] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:22:57.601065 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25f3d775-260b-4320-ba12-ccc076c1524a service neutron] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:22:57.601327 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25f3d775-260b-4320-ba12-ccc076c1524a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:22:57.602945 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-25f3d775-260b-4320-ba12-ccc076c1524a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:22:57.664236 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25f3d775-260b-4320-ba12-ccc076c1524a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:22:57.665665 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25f3d775-260b-4320-ba12-ccc076c1524a service neutron] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:22:57.675199 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1417/2803] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:22:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:23:00.761737 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8491be13-45d8-4bef-86d1-72af00cc46c4 service nova] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:00.763234 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-8491be13-45d8-4bef-86d1-72af00cc46c4 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:23:00.777578 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8491be13-45d8-4bef-86d1-72af00cc46c4 service nova] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:00.781804 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9237a71d-b77a-4067-830c-2a04f0008372 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:00.782420 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9237a71d-b77a-4067-830c-2a04f0008372 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:00.782701 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9237a71d-b77a-4067-830c-2a04f0008372 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:00.783011 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9237a71d-b77a-4067-830c-2a04f0008372 admin admin] PATH_INFO: `/v3/users/d079e0264f0e40db9fc93b8f217a7a74` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:00.784173 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9237a71d-b77a-4067-830c-2a04f0008372 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d079e0264f0e40db9fc93b8f217a7a74)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:00.786067 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9237a71d-b77a-4067-830c-2a04f0008372 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:00.832032 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-9237a71d-b77a-4067-830c-2a04f0008372 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd079e0264f0e40db9fc93b8f217a7a74'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:23:00.832679 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-9237a71d-b77a-4067-830c-2a04f0008372 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd079e0264f0e40db9fc93b8f217a7a74'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:23:00.833272 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-9237a71d-b77a-4067-830c-2a04f0008372 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd079e0264f0e40db9fc93b8f217a7a74'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:23:00.845357 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-9237a71d-b77a-4067-830c-2a04f0008372 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd079e0264f0e40db9fc93b8f217a7a74'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:23:00.852217 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-9237a71d-b77a-4067-830c-2a04f0008372 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd079e0264f0e40db9fc93b8f217a7a74'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:23:00.865444 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1387/2804] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:23:00 2022] DELETE /identity/v3/users/d079e0264f0e40db9fc93b8f217a7a74 => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 05:23:01.032200 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25f3d775-260b-4320-ba12-ccc076c1524a service neutron] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:01.038382 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=100 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:23:01.042481 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25f3d775-260b-4320-ba12-ccc076c1524a service neutron] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:01.102152 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18f3e264-3316-47a3-b619-c5930e570fae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:01.102762 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18f3e264-3316-47a3-b619-c5930e570fae admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:01.103064 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18f3e264-3316-47a3-b619-c5930e570fae admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:01.103334 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18f3e264-3316-47a3-b619-c5930e570fae admin admin] PATH_INFO: `/v3/projects/bebd859c24cd4ed3b7e32a82de820b8a` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:01.104436 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18f3e264-3316-47a3-b619-c5930e570fae admin admin] RBAC: Authorizing `identity:delete_project(project_id=bebd859c24cd4ed3b7e32a82de820b8a)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:01.106316 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18f3e264-3316-47a3-b619-c5930e570fae admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:01.151779 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-18f3e264-3316-47a3-b619-c5930e570fae admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bebd859c24cd4ed3b7e32a82de820b8a', 'request_id': 'req-18f3e264-3316-47a3-b619-c5930e570fae'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:23:01.158834 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-18f3e264-3316-47a3-b619-c5930e570fae admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bebd859c24cd4ed3b7e32a82de820b8a', 'request_id': 'req-18f3e264-3316-47a3-b619-c5930e570fae'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:23:01.164812 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-18f3e264-3316-47a3-b619-c5930e570fae admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bebd859c24cd4ed3b7e32a82de820b8a', 'request_id': 'req-18f3e264-3316-47a3-b619-c5930e570fae'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:23:01.175300 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-18f3e264-3316-47a3-b619-c5930e570fae admin admin] The token cache is being invalidate because project bebd859c24cd4ed3b7e32a82de820b8a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81625) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:23:01.175581 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-18f3e264-3316-47a3-b619-c5930e570fae admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:23:01.176192 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-18f3e264-3316-47a3-b619-c5930e570fae admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:23:01.177627 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1418/2805] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:23:01 2022] DELETE /identity/v3/projects/bebd859c24cd4ed3b7e32a82de820b8a => generated 0 bytes in 146 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:23:01.182663 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e14f1a1-19fa-475c-b80c-99ff81e0fe97 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:01.183015 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e14f1a1-19fa-475c-b80c-99ff81e0fe97 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:01.183286 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e14f1a1-19fa-475c-b80c-99ff81e0fe97 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:01.203000 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-6e14f1a1-19fa-475c-b80c-99ff81e0fe97 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:23:01.204177 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-6e14f1a1-19fa-475c-b80c-99ff81e0fe97 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:23:01.253450 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:23:01.269629 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1388/2806] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:23:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:23:01.272493 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dddd61bb-25ac-4ef0-a013-dfd9d00e4d10 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:01.272778 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dddd61bb-25ac-4ef0-a013-dfd9d00e4d10 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:01.273085 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dddd61bb-25ac-4ef0-a013-dfd9d00e4d10 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:01.292298 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-dddd61bb-25ac-4ef0-a013-dfd9d00e4d10 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:23:01.293495 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-dddd61bb-25ac-4ef0-a013-dfd9d00e4d10 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:23:01.301797 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1419/2807] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:23:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:23:01.305846 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e14f1a1-19fa-475c-b80c-99ff81e0fe97 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:01.315909 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e14f1a1-19fa-475c-b80c-99ff81e0fe97 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:01.320365 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aea8361-1b26-4c1d-9843-6e11e92e15c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:01.320944 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aea8361-1b26-4c1d-9843-6e11e92e15c7 admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:01.321238 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aea8361-1b26-4c1d-9843-6e11e92e15c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:01.321488 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aea8361-1b26-4c1d-9843-6e11e92e15c7 admin admin] PATH_INFO: `/v3/domains` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:01.322121 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aea8361-1b26-4c1d-9843-6e11e92e15c7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81624) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:23:01.322421 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aea8361-1b26-4c1d-9843-6e11e92e15c7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:01.324020 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aea8361-1b26-4c1d-9843-6e11e92e15c7 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:01.335267 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1389/2808] 10.222.0.77 () {58 vars in 1153 bytes} [Tue Aug 30 05:23:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:23:01.339676 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dddd61bb-25ac-4ef0-a013-dfd9d00e4d10 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:01.349250 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dddd61bb-25ac-4ef0-a013-dfd9d00e4d10 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:01.353865 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34cbf9bd-84da-4aad-b024-b5a0e25fd889 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:01.354442 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34cbf9bd-84da-4aad-b024-b5a0e25fd889 admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:01.354716 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34cbf9bd-84da-4aad-b024-b5a0e25fd889 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:01.355016 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34cbf9bd-84da-4aad-b024-b5a0e25fd889 admin admin] PATH_INFO: `/v3/projects` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:01.355641 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34cbf9bd-84da-4aad-b024-b5a0e25fd889 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:01.357526 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34cbf9bd-84da-4aad-b024-b5a0e25fd889 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:01.393204 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1420/2809] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:23:01 2022] POST /identity/v3/projects => generated 403 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:23:01.397174 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aea8361-1b26-4c1d-9843-6e11e92e15c7 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:01.407160 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aea8361-1b26-4c1d-9843-6e11e92e15c7 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:01.473144 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fdb0fbf-b2bf-4fa0-b1f0-14d064802e0f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:01.473775 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fdb0fbf-b2bf-4fa0-b1f0-14d064802e0f admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:01.474053 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fdb0fbf-b2bf-4fa0-b1f0-14d064802e0f admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:01.474322 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fdb0fbf-b2bf-4fa0-b1f0-14d064802e0f admin admin] PATH_INFO: `/v3/users` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:01.474990 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fdb0fbf-b2bf-4fa0-b1f0-14d064802e0f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:01.476893 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fdb0fbf-b2bf-4fa0-b1f0-14d064802e0f admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:01.500465 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1390/2810] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:23:01 2022] POST /identity/v3/users => generated 302 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:23:01.504577 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34cbf9bd-84da-4aad-b024-b5a0e25fd889 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:01.515055 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34cbf9bd-84da-4aad-b024-b5a0e25fd889 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:01.519621 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83211149-d556-4cff-92a1-8544230fcd09 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:01.520234 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83211149-d556-4cff-92a1-8544230fcd09 admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:01.520512 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83211149-d556-4cff-92a1-8544230fcd09 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:01.520777 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83211149-d556-4cff-92a1-8544230fcd09 admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:01.521400 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83211149-d556-4cff-92a1-8544230fcd09 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:01.523018 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83211149-d556-4cff-92a1-8544230fcd09 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:01.533983 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1421/2811] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:23:01 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:23:01.537870 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fdb0fbf-b2bf-4fa0-b1f0-14d064802e0f admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:01.547963 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fdb0fbf-b2bf-4fa0-b1f0-14d064802e0f admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:01.553602 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7d14ee8-49ef-43d8-91a0-cd5952f9a44f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:01.554402 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7d14ee8-49ef-43d8-91a0-cd5952f9a44f admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:01.554671 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7d14ee8-49ef-43d8-91a0-cd5952f9a44f admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:01.554980 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7d14ee8-49ef-43d8-91a0-cd5952f9a44f admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:01.555614 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7d14ee8-49ef-43d8-91a0-cd5952f9a44f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:01.557389 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7d14ee8-49ef-43d8-91a0-cd5952f9a44f admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:01.568165 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1391/2812] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:23:01 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:23:01.572388 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83211149-d556-4cff-92a1-8544230fcd09 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:01.582273 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83211149-d556-4cff-92a1-8544230fcd09 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:01.586753 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c6bbe06-ca1e-4739-860f-31a7d0777388 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:01.587426 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c6bbe06-ca1e-4739-860f-31a7d0777388 admin admin] REQUEST_METHOD: `PUT` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:01.587704 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c6bbe06-ca1e-4739-860f-31a7d0777388 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:01.587974 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c6bbe06-ca1e-4739-860f-31a7d0777388 admin admin] PATH_INFO: `/v3/projects/fefbc6ac66824b65a7220f99268557fe/users/a47164578b5c4de1827c237f7b4b91bf/roles/6f03d86980c34c36b44a8d6764ff4fda` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:01.602886 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c6bbe06-ca1e-4739-860f-31a7d0777388 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fefbc6ac66824b65a7220f99268557fe, user_id=a47164578b5c4de1827c237f7b4b91bf, role_id=6f03d86980c34c36b44a8d6764ff4fda)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:01.605498 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c6bbe06-ca1e-4739-860f-31a7d0777388 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:01.618772 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1422/2813] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:23:01 2022] PUT /identity/v3/projects/fefbc6ac66824b65a7220f99268557fe/users/a47164578b5c4de1827c237f7b4b91bf/roles/6f03d86980c34c36b44a8d6764ff4fda => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:23:01.629854 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4b3a28-4293-4c92-b91d-785b149a1851 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:01.630140 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4b3a28-4293-4c92-b91d-785b149a1851 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:01.630443 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4b3a28-4293-4c92-b91d-785b149a1851 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:01.656749 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-2e4b3a28-4293-4c92-b91d-785b149a1851 None None] MFA Rules not processed for user `a47164578b5c4de1827c237f7b4b91bf`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:23:01.658020 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-2e4b3a28-4293-4c92-b91d-785b149a1851 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:23:01.720253 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1392/2814] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:23:01 2022] POST /identity/v3/auth/tokens => generated 3280 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:23:01.725385 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c6bbe06-ca1e-4739-860f-31a7d0777388 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:01.749948 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:23:01.788244 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c6bbe06-ca1e-4739-860f-31a7d0777388 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:01.792789 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33c42b88-c12f-42fe-99e3-13a64a3a553f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:01.793378 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33c42b88-c12f-42fe-99e3-13a64a3a553f admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:01.793655 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33c42b88-c12f-42fe-99e3-13a64a3a553f admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:01.793924 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33c42b88-c12f-42fe-99e3-13a64a3a553f admin admin] PATH_INFO: `/v3/projects` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:01.794547 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33c42b88-c12f-42fe-99e3-13a64a3a553f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:01.796483 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33c42b88-c12f-42fe-99e3-13a64a3a553f admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:01.821199 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1423/2815] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:23:01 2022] POST /identity/v3/projects => generated 403 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:23:01.825297 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e4b3a28-4293-4c92-b91d-785b149a1851 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:01.836191 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e4b3a28-4293-4c92-b91d-785b149a1851 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:01.882931 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:23:01.892236 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c26b0a25-84db-4b9b-b7f7-a55ce72c1b72 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:01.893359 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c26b0a25-84db-4b9b-b7f7-a55ce72c1b72 admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:01.893633 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c26b0a25-84db-4b9b-b7f7-a55ce72c1b72 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:01.893901 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c26b0a25-84db-4b9b-b7f7-a55ce72c1b72 admin admin] PATH_INFO: `/v3/users` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:01.894515 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c26b0a25-84db-4b9b-b7f7-a55ce72c1b72 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:01.896446 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c26b0a25-84db-4b9b-b7f7-a55ce72c1b72 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:01.917442 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1393/2816] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:23:01 2022] POST /identity/v3/users => generated 308 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:23:01.921348 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33c42b88-c12f-42fe-99e3-13a64a3a553f admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:01.931456 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33c42b88-c12f-42fe-99e3-13a64a3a553f admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:01.936013 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b907cf05-cbbb-4e08-8870-e80b768c457b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:01.936592 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b907cf05-cbbb-4e08-8870-e80b768c457b admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:01.936870 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b907cf05-cbbb-4e08-8870-e80b768c457b admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:01.937140 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b907cf05-cbbb-4e08-8870-e80b768c457b admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:01.937768 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b907cf05-cbbb-4e08-8870-e80b768c457b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:01.939375 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b907cf05-cbbb-4e08-8870-e80b768c457b admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:01.949602 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1424/2817] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:23:01 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:23:01.953627 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c26b0a25-84db-4b9b-b7f7-a55ce72c1b72 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:01.964220 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c26b0a25-84db-4b9b-b7f7-a55ce72c1b72 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:01.968761 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1e3736b-e64b-43f6-af43-8c8570d4b187 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:01.969389 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1e3736b-e64b-43f6-af43-8c8570d4b187 admin admin] REQUEST_METHOD: `PUT` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:01.969667 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1e3736b-e64b-43f6-af43-8c8570d4b187 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:01.969939 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1e3736b-e64b-43f6-af43-8c8570d4b187 admin admin] PATH_INFO: `/v3/projects/8ba4e6f194854145897621aac6ec3fee/users/f6c2021102254c9c9e3254ef9cc97f3c/roles/9d9d5d2c270146488249b7ed943280db` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:01.984757 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1e3736b-e64b-43f6-af43-8c8570d4b187 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8ba4e6f194854145897621aac6ec3fee, user_id=f6c2021102254c9c9e3254ef9cc97f3c, role_id=9d9d5d2c270146488249b7ed943280db)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:01.987507 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1e3736b-e64b-43f6-af43-8c8570d4b187 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:01.999393 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1394/2818] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:23:01 2022] PUT /identity/v3/projects/8ba4e6f194854145897621aac6ec3fee/users/f6c2021102254c9c9e3254ef9cc97f3c/roles/9d9d5d2c270146488249b7ed943280db => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:23:02.003365 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b907cf05-cbbb-4e08-8870-e80b768c457b admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:02.013802 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b907cf05-cbbb-4e08-8870-e80b768c457b admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:02.072614 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21a4fb32-f0c4-4830-942a-9dab13cab2ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:02.073271 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a4fb32-f0c4-4830-942a-9dab13cab2ca admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:02.073472 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a4fb32-f0c4-4830-942a-9dab13cab2ca admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:02.073737 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a4fb32-f0c4-4830-942a-9dab13cab2ca admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:02.074363 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21a4fb32-f0c4-4830-942a-9dab13cab2ca admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:02.075973 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21a4fb32-f0c4-4830-942a-9dab13cab2ca admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:02.086291 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1425/2819] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:23:02 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:23:02.090746 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1e3736b-e64b-43f6-af43-8c8570d4b187 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:02.100903 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1e3736b-e64b-43f6-af43-8c8570d4b187 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:02.105465 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39ea546b-a751-4549-9be5-aa1bdb81ab33 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:02.106096 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39ea546b-a751-4549-9be5-aa1bdb81ab33 admin admin] REQUEST_METHOD: `PUT` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:02.106364 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39ea546b-a751-4549-9be5-aa1bdb81ab33 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:02.106637 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39ea546b-a751-4549-9be5-aa1bdb81ab33 admin admin] PATH_INFO: `/v3/projects/8ba4e6f194854145897621aac6ec3fee/users/f6c2021102254c9c9e3254ef9cc97f3c/roles/9d9d5d2c270146488249b7ed943280db` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:02.108685 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39ea546b-a751-4549-9be5-aa1bdb81ab33 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8ba4e6f194854145897621aac6ec3fee, user_id=f6c2021102254c9c9e3254ef9cc97f3c, role_id=9d9d5d2c270146488249b7ed943280db)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:02.111323 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39ea546b-a751-4549-9be5-aa1bdb81ab33 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:02.121481 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1395/2820] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:23:02 2022] PUT /identity/v3/projects/8ba4e6f194854145897621aac6ec3fee/users/f6c2021102254c9c9e3254ef9cc97f3c/roles/9d9d5d2c270146488249b7ed943280db => generated 0 bytes in 31 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:23:02.131959 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b40faa-d3f9-4588-88ae-b2a1df169817 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:02.132245 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b40faa-d3f9-4588-88ae-b2a1df169817 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:02.132517 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b40faa-d3f9-4588-88ae-b2a1df169817 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:02.159925 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-a3b40faa-d3f9-4588-88ae-b2a1df169817 None None] MFA Rules not processed for user `f6c2021102254c9c9e3254ef9cc97f3c`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:23:02.161117 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-a3b40faa-d3f9-4588-88ae-b2a1df169817 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:23:02.228472 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1426/2821] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:23:02 2022] POST /identity/v3/auth/tokens => generated 3347 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:23:02.239148 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39ea546b-a751-4549-9be5-aa1bdb81ab33 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:02.240861 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-39ea546b-a751-4549-9be5-aa1bdb81ab33 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:23:02.298210 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39ea546b-a751-4549-9be5-aa1bdb81ab33 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:02.301873 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42860c24-6db9-456b-b42d-2fab697dad06 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '019a79b2affe478b8a30d1c538d0415c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:02.302462 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42860c24-6db9-456b-b42d-2fab697dad06 service cinder] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:02.302737 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42860c24-6db9-456b-b42d-2fab697dad06 service cinder] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:02.303035 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42860c24-6db9-456b-b42d-2fab697dad06 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:02.311790 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42860c24-6db9-456b-b42d-2fab697dad06 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:02.313196 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42860c24-6db9-456b-b42d-2fab697dad06 service cinder] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:02.323256 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1396/2822] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:23:02 2022] GET /identity/v3/auth/tokens => generated 3347 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:23:02.461725 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3b40faa-d3f9-4588-88ae-b2a1df169817 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:02.481346 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:23:02.524830 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3b40faa-d3f9-4588-88ae-b2a1df169817 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:02.529265 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cde503fa-1469-4b10-9a9e-69462e7aa274 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:02.529873 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cde503fa-1469-4b10-9a9e-69462e7aa274 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:02.530149 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cde503fa-1469-4b10-9a9e-69462e7aa274 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:02.530417 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cde503fa-1469-4b10-9a9e-69462e7aa274 admin admin] PATH_INFO: `/v3/users/a47164578b5c4de1827c237f7b4b91bf` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:02.531549 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cde503fa-1469-4b10-9a9e-69462e7aa274 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a47164578b5c4de1827c237f7b4b91bf)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:02.533478 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cde503fa-1469-4b10-9a9e-69462e7aa274 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:02.579382 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-cde503fa-1469-4b10-9a9e-69462e7aa274 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a47164578b5c4de1827c237f7b4b91bf'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:23:02.586445 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-cde503fa-1469-4b10-9a9e-69462e7aa274 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a47164578b5c4de1827c237f7b4b91bf'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:23:02.598482 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-cde503fa-1469-4b10-9a9e-69462e7aa274 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a47164578b5c4de1827c237f7b4b91bf'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:23:02.605191 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-cde503fa-1469-4b10-9a9e-69462e7aa274 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a47164578b5c4de1827c237f7b4b91bf'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:23:02.605816 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-cde503fa-1469-4b10-9a9e-69462e7aa274 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a47164578b5c4de1827c237f7b4b91bf'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:23:02.612043 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1427/2823] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:23:02 2022] DELETE /identity/v3/users/a47164578b5c4de1827c237f7b4b91bf => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:23:02.616546 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31b3bf7d-aea0-4ca1-9dfd-676fba8cb44f None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:02.616826 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31b3bf7d-aea0-4ca1-9dfd-676fba8cb44f None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:02.617096 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31b3bf7d-aea0-4ca1-9dfd-676fba8cb44f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:02.636045 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-31b3bf7d-aea0-4ca1-9dfd-676fba8cb44f None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:23:02.637227 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-31b3bf7d-aea0-4ca1-9dfd-676fba8cb44f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:23:02.695293 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1397/2824] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:23:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:23:02.705451 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cde503fa-1469-4b10-9a9e-69462e7aa274 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:02.706881 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-cde503fa-1469-4b10-9a9e-69462e7aa274 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:23:02.763775 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cde503fa-1469-4b10-9a9e-69462e7aa274 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:02.767289 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6136f68d-4fc5-412a-a87b-e6787a4ed855 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:02.767875 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6136f68d-4fc5-412a-a87b-e6787a4ed855 service neutron] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:02.768147 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6136f68d-4fc5-412a-a87b-e6787a4ed855 service neutron] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:02.768413 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6136f68d-4fc5-412a-a87b-e6787a4ed855 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:02.777084 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6136f68d-4fc5-412a-a87b-e6787a4ed855 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:02.778483 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6136f68d-4fc5-412a-a87b-e6787a4ed855 service neutron] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:02.787635 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1428/2825] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:23:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:23:03.051604 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31b3bf7d-aea0-4ca1-9dfd-676fba8cb44f None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:03.053082 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-31b3bf7d-aea0-4ca1-9dfd-676fba8cb44f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:23:03.065541 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31b3bf7d-aea0-4ca1-9dfd-676fba8cb44f None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:03.070444 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31ddff86-3f7a-4326-a77d-fba5aa4ce5b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:03.071105 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31ddff86-3f7a-4326-a77d-fba5aa4ce5b4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:03.071384 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31ddff86-3f7a-4326-a77d-fba5aa4ce5b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:03.071698 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31ddff86-3f7a-4326-a77d-fba5aa4ce5b4 admin admin] PATH_INFO: `/v3/projects/fefbc6ac66824b65a7220f99268557fe` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:03.072794 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31ddff86-3f7a-4326-a77d-fba5aa4ce5b4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fefbc6ac66824b65a7220f99268557fe)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:03.074921 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31ddff86-3f7a-4326-a77d-fba5aa4ce5b4 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:03.119120 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=92,keystone:INSERT=5,keystone:DELETE=3 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:23:03.123118 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-31ddff86-3f7a-4326-a77d-fba5aa4ce5b4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fefbc6ac66824b65a7220f99268557fe', 'request_id': 'req-31ddff86-3f7a-4326-a77d-fba5aa4ce5b4'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:23:03.131159 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-31ddff86-3f7a-4326-a77d-fba5aa4ce5b4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fefbc6ac66824b65a7220f99268557fe', 'request_id': 'req-31ddff86-3f7a-4326-a77d-fba5aa4ce5b4'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:23:03.137382 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-31ddff86-3f7a-4326-a77d-fba5aa4ce5b4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fefbc6ac66824b65a7220f99268557fe', 'request_id': 'req-31ddff86-3f7a-4326-a77d-fba5aa4ce5b4'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:23:03.147695 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-31ddff86-3f7a-4326-a77d-fba5aa4ce5b4 admin admin] The token cache is being invalidate because project fefbc6ac66824b65a7220f99268557fe was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81624) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:23:03.147979 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-31ddff86-3f7a-4326-a77d-fba5aa4ce5b4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:23:03.148612 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-31ddff86-3f7a-4326-a77d-fba5aa4ce5b4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:23:03.150075 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1398/2826] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:23:03 2022] DELETE /identity/v3/projects/fefbc6ac66824b65a7220f99268557fe => generated 0 bytes in 99 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:23:03.154114 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6136f68d-4fc5-412a-a87b-e6787a4ed855 service neutron] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:03.166010 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6136f68d-4fc5-412a-a87b-e6787a4ed855 service neutron] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:03.204144 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:23:03.225861 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7512cd04-4ada-43d4-897a-df49cdea10da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:03.226477 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7512cd04-4ada-43d4-897a-df49cdea10da admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:03.226757 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7512cd04-4ada-43d4-897a-df49cdea10da admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:03.227059 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7512cd04-4ada-43d4-897a-df49cdea10da admin admin] PATH_INFO: `/v3/users/f6c2021102254c9c9e3254ef9cc97f3c` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:03.228165 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7512cd04-4ada-43d4-897a-df49cdea10da admin admin] RBAC: Authorizing `identity:delete_user(user_id=f6c2021102254c9c9e3254ef9cc97f3c)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:03.230074 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7512cd04-4ada-43d4-897a-df49cdea10da admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:03.275011 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-7512cd04-4ada-43d4-897a-df49cdea10da admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f6c2021102254c9c9e3254ef9cc97f3c'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:23:03.282070 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-7512cd04-4ada-43d4-897a-df49cdea10da admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f6c2021102254c9c9e3254ef9cc97f3c'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:23:03.294476 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-7512cd04-4ada-43d4-897a-df49cdea10da admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f6c2021102254c9c9e3254ef9cc97f3c'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:23:03.301206 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-7512cd04-4ada-43d4-897a-df49cdea10da admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f6c2021102254c9c9e3254ef9cc97f3c'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:23:03.301828 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-7512cd04-4ada-43d4-897a-df49cdea10da admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f6c2021102254c9c9e3254ef9cc97f3c'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:23:03.307509 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1429/2827] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:23:03 2022] DELETE /identity/v3/users/f6c2021102254c9c9e3254ef9cc97f3c => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:23:03.567701 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31ddff86-3f7a-4326-a77d-fba5aa4ce5b4 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:03.569227 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-31ddff86-3f7a-4326-a77d-fba5aa4ce5b4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:23:03.583025 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31ddff86-3f7a-4326-a77d-fba5aa4ce5b4 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:03.639636 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99106857-353a-4fb9-8c41-0ae1c420eb5c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:03.640248 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99106857-353a-4fb9-8c41-0ae1c420eb5c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:03.640524 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99106857-353a-4fb9-8c41-0ae1c420eb5c admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:03.640787 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99106857-353a-4fb9-8c41-0ae1c420eb5c admin admin] PATH_INFO: `/v3/projects/8ba4e6f194854145897621aac6ec3fee` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:03.641865 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99106857-353a-4fb9-8c41-0ae1c420eb5c admin admin] RBAC: Authorizing `identity:delete_project(project_id=8ba4e6f194854145897621aac6ec3fee)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:03.644338 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99106857-353a-4fb9-8c41-0ae1c420eb5c admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:03.690758 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-99106857-353a-4fb9-8c41-0ae1c420eb5c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8ba4e6f194854145897621aac6ec3fee', 'request_id': 'req-99106857-353a-4fb9-8c41-0ae1c420eb5c'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:23:03.699091 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-99106857-353a-4fb9-8c41-0ae1c420eb5c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8ba4e6f194854145897621aac6ec3fee', 'request_id': 'req-99106857-353a-4fb9-8c41-0ae1c420eb5c'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:23:03.705980 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-99106857-353a-4fb9-8c41-0ae1c420eb5c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8ba4e6f194854145897621aac6ec3fee', 'request_id': 'req-99106857-353a-4fb9-8c41-0ae1c420eb5c'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:23:03.716312 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-99106857-353a-4fb9-8c41-0ae1c420eb5c admin admin] The token cache is being invalidate because project 8ba4e6f194854145897621aac6ec3fee was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81624) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:23:03.716596 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-99106857-353a-4fb9-8c41-0ae1c420eb5c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:23:03.717215 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-99106857-353a-4fb9-8c41-0ae1c420eb5c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:23:03.718656 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1399/2828] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:23:03 2022] DELETE /identity/v3/projects/8ba4e6f194854145897621aac6ec3fee => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:23:03.723465 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29bbd428-3b70-4c7a-9ae0-1c8e9d84e343 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:03.723751 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29bbd428-3b70-4c7a-9ae0-1c8e9d84e343 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:03.724018 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29bbd428-3b70-4c7a-9ae0-1c8e9d84e343 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:03.743233 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-29bbd428-3b70-4c7a-9ae0-1c8e9d84e343 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:23:03.744461 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-29bbd428-3b70-4c7a-9ae0-1c8e9d84e343 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:23:03.802201 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1430/2829] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:23:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:23:03.810056 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0500f2cf-df4f-4f9e-b2f7-5e135b2492f0 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:03.810279 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0500f2cf-df4f-4f9e-b2f7-5e135b2492f0 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:03.810559 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0500f2cf-df4f-4f9e-b2f7-5e135b2492f0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:03.830696 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-0500f2cf-df4f-4f9e-b2f7-5e135b2492f0 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:23:03.831918 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-0500f2cf-df4f-4f9e-b2f7-5e135b2492f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:23:03.840274 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1400/2830] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:23:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:23:03.844267 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29bbd428-3b70-4c7a-9ae0-1c8e9d84e343 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:03.853910 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29bbd428-3b70-4c7a-9ae0-1c8e9d84e343 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:03.858357 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-711d82e7-4dbe-4076-9d3c-5192e9f65cd3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:03.858957 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-711d82e7-4dbe-4076-9d3c-5192e9f65cd3 admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:03.859235 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-711d82e7-4dbe-4076-9d3c-5192e9f65cd3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:03.859501 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-711d82e7-4dbe-4076-9d3c-5192e9f65cd3 admin admin] PATH_INFO: `/v3/domains` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:03.860133 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-711d82e7-4dbe-4076-9d3c-5192e9f65cd3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:23:03.860427 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-711d82e7-4dbe-4076-9d3c-5192e9f65cd3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:03.862019 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-711d82e7-4dbe-4076-9d3c-5192e9f65cd3 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:03.873565 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1431/2831] 10.222.0.77 () {58 vars in 1153 bytes} [Tue Aug 30 05:23:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 05:23:03.878015 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0500f2cf-df4f-4f9e-b2f7-5e135b2492f0 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:03.887009 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0500f2cf-df4f-4f9e-b2f7-5e135b2492f0 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:03.891489 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b6c6937-14fc-4c72-ae4c-498b049f21c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:03.892051 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b6c6937-14fc-4c72-ae4c-498b049f21c1 admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:03.892324 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b6c6937-14fc-4c72-ae4c-498b049f21c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:03.893216 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b6c6937-14fc-4c72-ae4c-498b049f21c1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:03.893836 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b6c6937-14fc-4c72-ae4c-498b049f21c1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:03.895735 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b6c6937-14fc-4c72-ae4c-498b049f21c1 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:03.921419 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1401/2832] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:23:03 2022] POST /identity/v3/projects => generated 381 bytes in 44 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 05:23:03.925774 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-711d82e7-4dbe-4076-9d3c-5192e9f65cd3 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:03.935934 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-711d82e7-4dbe-4076-9d3c-5192e9f65cd3 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:03.980173 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:23:03.994108 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e3eb296-210a-4a42-933d-a2aa5523a8e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:03.994701 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e3eb296-210a-4a42-933d-a2aa5523a8e9 admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:03.995003 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e3eb296-210a-4a42-933d-a2aa5523a8e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:03.995271 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e3eb296-210a-4a42-933d-a2aa5523a8e9 admin admin] PATH_INFO: `/v3/users` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:03.995901 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e3eb296-210a-4a42-933d-a2aa5523a8e9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:03.997785 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e3eb296-210a-4a42-933d-a2aa5523a8e9 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:04.017540 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1432/2833] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:23:03 2022] POST /identity/v3/users => generated 291 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 05:23:04.021719 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b6c6937-14fc-4c72-ae4c-498b049f21c1 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:04.031997 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b6c6937-14fc-4c72-ae4c-498b049f21c1 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:04.036541 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2fe9610-30f2-4093-a7cf-cc22c6c687a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:04.037121 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2fe9610-30f2-4093-a7cf-cc22c6c687a9 admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:04.037396 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2fe9610-30f2-4093-a7cf-cc22c6c687a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:04.037664 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2fe9610-30f2-4093-a7cf-cc22c6c687a9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:04.038289 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2fe9610-30f2-4093-a7cf-cc22c6c687a9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:04.039930 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2fe9610-30f2-4093-a7cf-cc22c6c687a9 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:04.050917 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1402/2834] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:23:04 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:23:04.054684 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e3eb296-210a-4a42-933d-a2aa5523a8e9 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:04.065248 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e3eb296-210a-4a42-933d-a2aa5523a8e9 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:04.069591 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8146f184-21df-4577-a75e-e95830bccdb1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:04.070170 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8146f184-21df-4577-a75e-e95830bccdb1 admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:04.070446 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8146f184-21df-4577-a75e-e95830bccdb1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:04.070711 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8146f184-21df-4577-a75e-e95830bccdb1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:04.071373 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8146f184-21df-4577-a75e-e95830bccdb1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:04.072941 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8146f184-21df-4577-a75e-e95830bccdb1 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:04.083225 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1433/2835] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:23:04 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:23:04.087111 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2fe9610-30f2-4093-a7cf-cc22c6c687a9 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:04.097461 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2fe9610-30f2-4093-a7cf-cc22c6c687a9 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:04.101967 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8fa1dbf-6e30-444c-bdb5-efac8e1a8c0a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:04.102606 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8fa1dbf-6e30-444c-bdb5-efac8e1a8c0a admin admin] REQUEST_METHOD: `PUT` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:04.102912 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8fa1dbf-6e30-444c-bdb5-efac8e1a8c0a admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:04.103185 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8fa1dbf-6e30-444c-bdb5-efac8e1a8c0a admin admin] PATH_INFO: `/v3/projects/585083dd7f47448ca405cd01d0b1ab0b/users/0e974ba183314af7b7c66444661f7ba9/roles/6f03d86980c34c36b44a8d6764ff4fda` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:04.117773 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8fa1dbf-6e30-444c-bdb5-efac8e1a8c0a admin admin] RBAC: Authorizing `identity:create_grant(project_id=585083dd7f47448ca405cd01d0b1ab0b, user_id=0e974ba183314af7b7c66444661f7ba9, role_id=6f03d86980c34c36b44a8d6764ff4fda)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:04.120452 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8fa1dbf-6e30-444c-bdb5-efac8e1a8c0a admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:04.128893 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=85,keystone:DELETE=11,keystone:INSERT=4 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:23:04.135276 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1403/2836] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:23:04 2022] PUT /identity/v3/projects/585083dd7f47448ca405cd01d0b1ab0b/users/0e974ba183314af7b7c66444661f7ba9/roles/6f03d86980c34c36b44a8d6764ff4fda => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:23:04.146601 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-710f4852-96d1-4869-8137-4402af6b5915 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:04.146930 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-710f4852-96d1-4869-8137-4402af6b5915 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:04.147204 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-710f4852-96d1-4869-8137-4402af6b5915 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:04.174495 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-710f4852-96d1-4869-8137-4402af6b5915 None None] MFA Rules not processed for user `0e974ba183314af7b7c66444661f7ba9`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:23:04.175734 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-710f4852-96d1-4869-8137-4402af6b5915 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:23:04.238167 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1434/2837] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:23:04 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:23:04.242606 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8fa1dbf-6e30-444c-bdb5-efac8e1a8c0a admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:04.309261 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8fa1dbf-6e30-444c-bdb5-efac8e1a8c0a admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:04.315553 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-082f1f2f-89fa-4005-9f70-6c951c426b05 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:04.316120 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-082f1f2f-89fa-4005-9f70-6c951c426b05 admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:04.316394 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-082f1f2f-89fa-4005-9f70-6c951c426b05 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:04.316659 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-082f1f2f-89fa-4005-9f70-6c951c426b05 admin admin] PATH_INFO: `/v3/projects` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:04.317323 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-082f1f2f-89fa-4005-9f70-6c951c426b05 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:04.319311 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-082f1f2f-89fa-4005-9f70-6c951c426b05 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:04.351757 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1404/2838] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:23:04 2022] POST /identity/v3/projects => generated 381 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:23:04.353249 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-710f4852-96d1-4869-8137-4402af6b5915 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:04.365293 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-710f4852-96d1-4869-8137-4402af6b5915 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:04.423496 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36a03d6d-0654-440c-90f9-23e77df19b53 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:04.424663 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a03d6d-0654-440c-90f9-23e77df19b53 admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:04.424940 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a03d6d-0654-440c-90f9-23e77df19b53 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:04.425210 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a03d6d-0654-440c-90f9-23e77df19b53 admin admin] PATH_INFO: `/v3/users` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:04.425841 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36a03d6d-0654-440c-90f9-23e77df19b53 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:04.427783 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36a03d6d-0654-440c-90f9-23e77df19b53 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:04.448262 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1435/2839] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:23:04 2022] POST /identity/v3/users => generated 297 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:23:04.452268 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-082f1f2f-89fa-4005-9f70-6c951c426b05 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:04.463333 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-082f1f2f-89fa-4005-9f70-6c951c426b05 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:04.472330 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c15e9878-22fa-4301-9632-f5e65c5026c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:04.472957 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c15e9878-22fa-4301-9632-f5e65c5026c4 admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:04.473232 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c15e9878-22fa-4301-9632-f5e65c5026c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:04.473497 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c15e9878-22fa-4301-9632-f5e65c5026c4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:04.474131 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c15e9878-22fa-4301-9632-f5e65c5026c4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:04.475826 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c15e9878-22fa-4301-9632-f5e65c5026c4 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:04.487154 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1405/2840] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:23:04 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 05:23:04.491398 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36a03d6d-0654-440c-90f9-23e77df19b53 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:04.501085 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36a03d6d-0654-440c-90f9-23e77df19b53 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:04.505631 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b377faa-ac45-4bcf-9880-e8a40dd482c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:04.506259 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b377faa-ac45-4bcf-9880-e8a40dd482c5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:04.506534 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b377faa-ac45-4bcf-9880-e8a40dd482c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:04.506801 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b377faa-ac45-4bcf-9880-e8a40dd482c5 admin admin] PATH_INFO: `/v3/projects/0022bcc23b2547a8b15709b0e25b4c3b/users/51b8f091bfff466789f7065ef89e20a8/roles/9d9d5d2c270146488249b7ed943280db` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:04.520860 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b377faa-ac45-4bcf-9880-e8a40dd482c5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0022bcc23b2547a8b15709b0e25b4c3b, user_id=51b8f091bfff466789f7065ef89e20a8, role_id=9d9d5d2c270146488249b7ed943280db)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:04.523495 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b377faa-ac45-4bcf-9880-e8a40dd482c5 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:04.536221 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1436/2841] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:23:04 2022] PUT /identity/v3/projects/0022bcc23b2547a8b15709b0e25b4c3b/users/51b8f091bfff466789f7065ef89e20a8/roles/9d9d5d2c270146488249b7ed943280db => generated 0 bytes in 45 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:23:04.539997 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c15e9878-22fa-4301-9632-f5e65c5026c4 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:04.551089 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c15e9878-22fa-4301-9632-f5e65c5026c4 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:04.605972 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd80bc43-488e-48e9-bf9e-89f0de8d5447 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:04.606493 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd80bc43-488e-48e9-bf9e-89f0de8d5447 admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:04.606770 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd80bc43-488e-48e9-bf9e-89f0de8d5447 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:04.607069 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd80bc43-488e-48e9-bf9e-89f0de8d5447 admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:04.607695 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd80bc43-488e-48e9-bf9e-89f0de8d5447 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:04.609274 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd80bc43-488e-48e9-bf9e-89f0de8d5447 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:04.619409 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1406/2842] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:23:04 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:23:04.623502 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b377faa-ac45-4bcf-9880-e8a40dd482c5 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:04.633880 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b377faa-ac45-4bcf-9880-e8a40dd482c5 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:04.638639 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be37a7ce-89bf-4a08-9617-e24f6e590da6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:04.639312 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be37a7ce-89bf-4a08-9617-e24f6e590da6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:04.639591 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be37a7ce-89bf-4a08-9617-e24f6e590da6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:04.639858 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be37a7ce-89bf-4a08-9617-e24f6e590da6 admin admin] PATH_INFO: `/v3/projects/0022bcc23b2547a8b15709b0e25b4c3b/users/51b8f091bfff466789f7065ef89e20a8/roles/9d9d5d2c270146488249b7ed943280db` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:04.641815 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be37a7ce-89bf-4a08-9617-e24f6e590da6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0022bcc23b2547a8b15709b0e25b4c3b, user_id=51b8f091bfff466789f7065ef89e20a8, role_id=9d9d5d2c270146488249b7ed943280db)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:04.644493 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be37a7ce-89bf-4a08-9617-e24f6e590da6 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:04.654526 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1437/2843] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:23:04 2022] PUT /identity/v3/projects/0022bcc23b2547a8b15709b0e25b4c3b/users/51b8f091bfff466789f7065ef89e20a8/roles/9d9d5d2c270146488249b7ed943280db => generated 0 bytes in 31 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:23:04.664853 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84c3ceba-5938-4388-b314-8baee2b2fdc1 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:04.665141 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84c3ceba-5938-4388-b314-8baee2b2fdc1 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:04.665414 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84c3ceba-5938-4388-b314-8baee2b2fdc1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:04.694108 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-84c3ceba-5938-4388-b314-8baee2b2fdc1 None None] MFA Rules not processed for user `51b8f091bfff466789f7065ef89e20a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:23:04.695333 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-84c3ceba-5938-4388-b314-8baee2b2fdc1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:23:04.752684 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:23:04.763440 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1407/2844] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:23:04 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:23:04.775241 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be37a7ce-89bf-4a08-9617-e24f6e590da6 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:04.776946 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-be37a7ce-89bf-4a08-9617-e24f6e590da6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:23:04.788477 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:23:04.839092 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be37a7ce-89bf-4a08-9617-e24f6e590da6 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:04.843846 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4432017-698a-45d8-93a3-5aa4780d6cce service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '019a79b2affe478b8a30d1c538d0415c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:04.844443 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4432017-698a-45d8-93a3-5aa4780d6cce service cinder] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:04.844719 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4432017-698a-45d8-93a3-5aa4780d6cce service cinder] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:04.844987 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4432017-698a-45d8-93a3-5aa4780d6cce service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:04.854679 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4432017-698a-45d8-93a3-5aa4780d6cce service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:04.856179 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4432017-698a-45d8-93a3-5aa4780d6cce service cinder] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:04.865949 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1438/2845] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:23:04 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:23:04.940128 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84c3ceba-5938-4388-b314-8baee2b2fdc1 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:04.949626 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84c3ceba-5938-4388-b314-8baee2b2fdc1 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:04.953133 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b46e7a24-8878-47ee-91b0-c027eccd71e1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '019a79b2affe478b8a30d1c538d0415c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:04.953711 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b46e7a24-8878-47ee-91b0-c027eccd71e1 service cinder] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:04.954010 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b46e7a24-8878-47ee-91b0-c027eccd71e1 service cinder] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:04.954271 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b46e7a24-8878-47ee-91b0-c027eccd71e1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:05.010375 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b46e7a24-8878-47ee-91b0-c027eccd71e1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:05.011834 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b46e7a24-8878-47ee-91b0-c027eccd71e1 service cinder] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:05.021224 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1408/2846] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:23:04 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 82 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:23:13.796404 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4432017-698a-45d8-93a3-5aa4780d6cce service cinder] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:13.860209 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4432017-698a-45d8-93a3-5aa4780d6cce service cinder] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:13.865160 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3994cdb5-24ac-4fb1-95de-19d3b71067b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:13.865775 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3994cdb5-24ac-4fb1-95de-19d3b71067b1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:13.866052 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3994cdb5-24ac-4fb1-95de-19d3b71067b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:13.866317 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3994cdb5-24ac-4fb1-95de-19d3b71067b1 admin admin] PATH_INFO: `/v3/users/0e974ba183314af7b7c66444661f7ba9` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:13.867442 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3994cdb5-24ac-4fb1-95de-19d3b71067b1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0e974ba183314af7b7c66444661f7ba9)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:13.869336 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3994cdb5-24ac-4fb1-95de-19d3b71067b1 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:13.915374 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-3994cdb5-24ac-4fb1-95de-19d3b71067b1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0e974ba183314af7b7c66444661f7ba9'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:23:13.922272 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-3994cdb5-24ac-4fb1-95de-19d3b71067b1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0e974ba183314af7b7c66444661f7ba9'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:23:13.937120 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-3994cdb5-24ac-4fb1-95de-19d3b71067b1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0e974ba183314af7b7c66444661f7ba9'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:23:13.944633 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-3994cdb5-24ac-4fb1-95de-19d3b71067b1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0e974ba183314af7b7c66444661f7ba9'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:23:13.945288 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-3994cdb5-24ac-4fb1-95de-19d3b71067b1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0e974ba183314af7b7c66444661f7ba9'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:23:13.951272 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1439/2847] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:23:13 2022] DELETE /identity/v3/users/0e974ba183314af7b7c66444661f7ba9 => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:23:13.955668 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a75178f6-ec97-4e39-82ff-2d05942d3d23 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:13.956164 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a75178f6-ec97-4e39-82ff-2d05942d3d23 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:13.956349 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a75178f6-ec97-4e39-82ff-2d05942d3d23 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:13.976935 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-a75178f6-ec97-4e39-82ff-2d05942d3d23 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:23:13.978135 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-a75178f6-ec97-4e39-82ff-2d05942d3d23 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:23:14.035204 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1409/2848] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:23:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:23:14.044733 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3994cdb5-24ac-4fb1-95de-19d3b71067b1 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:14.046281 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-3994cdb5-24ac-4fb1-95de-19d3b71067b1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:23:14.079322 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:23:14.106067 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3994cdb5-24ac-4fb1-95de-19d3b71067b1 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:14.109671 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eba16ea-644f-4311-b7a6-621d3f089228 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:14.110259 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eba16ea-644f-4311-b7a6-621d3f089228 service neutron] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:14.110534 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eba16ea-644f-4311-b7a6-621d3f089228 service neutron] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:14.110801 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eba16ea-644f-4311-b7a6-621d3f089228 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:14.120451 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eba16ea-644f-4311-b7a6-621d3f089228 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:14.121871 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eba16ea-644f-4311-b7a6-621d3f089228 service neutron] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:14.131238 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1440/2849] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:23:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 05:23:14.391450 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a75178f6-ec97-4e39-82ff-2d05942d3d23 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:14.392954 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-a75178f6-ec97-4e39-82ff-2d05942d3d23 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:23:14.406791 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a75178f6-ec97-4e39-82ff-2d05942d3d23 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:14.412413 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb183776-ea67-4784-9c4c-40c136315c6f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:14.413286 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb183776-ea67-4784-9c4c-40c136315c6f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:14.413644 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb183776-ea67-4784-9c4c-40c136315c6f admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:14.414016 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb183776-ea67-4784-9c4c-40c136315c6f admin admin] PATH_INFO: `/v3/projects/585083dd7f47448ca405cd01d0b1ab0b` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:14.415296 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb183776-ea67-4784-9c4c-40c136315c6f admin admin] RBAC: Authorizing `identity:delete_project(project_id=585083dd7f47448ca405cd01d0b1ab0b)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:14.417288 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb183776-ea67-4784-9c4c-40c136315c6f admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:14.466678 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-eb183776-ea67-4784-9c4c-40c136315c6f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '585083dd7f47448ca405cd01d0b1ab0b', 'request_id': 'req-eb183776-ea67-4784-9c4c-40c136315c6f'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:23:14.475116 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-eb183776-ea67-4784-9c4c-40c136315c6f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '585083dd7f47448ca405cd01d0b1ab0b', 'request_id': 'req-eb183776-ea67-4784-9c4c-40c136315c6f'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:23:14.481289 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-eb183776-ea67-4784-9c4c-40c136315c6f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '585083dd7f47448ca405cd01d0b1ab0b', 'request_id': 'req-eb183776-ea67-4784-9c4c-40c136315c6f'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:23:14.491429 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-eb183776-ea67-4784-9c4c-40c136315c6f admin admin] The token cache is being invalidate because project 585083dd7f47448ca405cd01d0b1ab0b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81624) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:23:14.492331 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-eb183776-ea67-4784-9c4c-40c136315c6f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:23:14.492964 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-eb183776-ea67-4784-9c4c-40c136315c6f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:23:14.494430 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1410/2850] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:23:14 2022] DELETE /identity/v3/projects/585083dd7f47448ca405cd01d0b1ab0b => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:23:14.498617 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eba16ea-644f-4311-b7a6-621d3f089228 service neutron] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:14.508125 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eba16ea-644f-4311-b7a6-621d3f089228 service neutron] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:14.565865 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a792cb69-0cea-4670-979a-438bd37dc786 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:14.566477 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a792cb69-0cea-4670-979a-438bd37dc786 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:14.566755 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a792cb69-0cea-4670-979a-438bd37dc786 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:14.567052 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a792cb69-0cea-4670-979a-438bd37dc786 admin admin] PATH_INFO: `/v3/users/51b8f091bfff466789f7065ef89e20a8` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:14.568180 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a792cb69-0cea-4670-979a-438bd37dc786 admin admin] RBAC: Authorizing `identity:delete_user(user_id=51b8f091bfff466789f7065ef89e20a8)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:14.570093 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a792cb69-0cea-4670-979a-438bd37dc786 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:14.615144 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-a792cb69-0cea-4670-979a-438bd37dc786 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '51b8f091bfff466789f7065ef89e20a8'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:23:14.622028 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-a792cb69-0cea-4670-979a-438bd37dc786 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '51b8f091bfff466789f7065ef89e20a8'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:23:14.633605 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-a792cb69-0cea-4670-979a-438bd37dc786 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '51b8f091bfff466789f7065ef89e20a8'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:23:14.640327 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-a792cb69-0cea-4670-979a-438bd37dc786 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '51b8f091bfff466789f7065ef89e20a8'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:23:14.640963 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-a792cb69-0cea-4670-979a-438bd37dc786 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '51b8f091bfff466789f7065ef89e20a8'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:23:14.647166 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1441/2851] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:23:14 2022] DELETE /identity/v3/users/51b8f091bfff466789f7065ef89e20a8 => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:23:14.911582 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb183776-ea67-4784-9c4c-40c136315c6f admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:14.913031 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-eb183776-ea67-4784-9c4c-40c136315c6f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:23:14.930977 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb183776-ea67-4784-9c4c-40c136315c6f admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:14.944006 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:23:14.990259 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b14c82a7-40c5-4c80-88f3-81c8664798f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:14.990922 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b14c82a7-40c5-4c80-88f3-81c8664798f6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:14.991098 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b14c82a7-40c5-4c80-88f3-81c8664798f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:14.991366 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b14c82a7-40c5-4c80-88f3-81c8664798f6 admin admin] PATH_INFO: `/v3/projects/0022bcc23b2547a8b15709b0e25b4c3b` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:14.992445 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b14c82a7-40c5-4c80-88f3-81c8664798f6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0022bcc23b2547a8b15709b0e25b4c3b)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:14.994316 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b14c82a7-40c5-4c80-88f3-81c8664798f6 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:15.039190 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-b14c82a7-40c5-4c80-88f3-81c8664798f6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0022bcc23b2547a8b15709b0e25b4c3b', 'request_id': 'req-b14c82a7-40c5-4c80-88f3-81c8664798f6'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:23:15.046489 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-b14c82a7-40c5-4c80-88f3-81c8664798f6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0022bcc23b2547a8b15709b0e25b4c3b', 'request_id': 'req-b14c82a7-40c5-4c80-88f3-81c8664798f6'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:23:15.053057 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-b14c82a7-40c5-4c80-88f3-81c8664798f6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0022bcc23b2547a8b15709b0e25b4c3b', 'request_id': 'req-b14c82a7-40c5-4c80-88f3-81c8664798f6'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:23:15.063286 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-b14c82a7-40c5-4c80-88f3-81c8664798f6 admin admin] The token cache is being invalidate because project 0022bcc23b2547a8b15709b0e25b4c3b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81624) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:23:15.063575 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-b14c82a7-40c5-4c80-88f3-81c8664798f6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:23:15.064191 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-b14c82a7-40c5-4c80-88f3-81c8664798f6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:23:15.066130 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1411/2852] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:23:14 2022] DELETE /identity/v3/projects/0022bcc23b2547a8b15709b0e25b4c3b => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:23:15.070950 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc62f13-c28d-4de2-a1c4-080211acc83d None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:15.071458 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc62f13-c28d-4de2-a1c4-080211acc83d None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:15.071561 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc62f13-c28d-4de2-a1c4-080211acc83d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:15.091432 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-5cc62f13-c28d-4de2-a1c4-080211acc83d None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:23:15.092623 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-5cc62f13-c28d-4de2-a1c4-080211acc83d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:23:15.135518 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:23:15.151523 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1442/2853] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:23:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:23:15.212691 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8709f98c-1249-47a7-9472-1c8c0095fd7b None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:15.213228 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8709f98c-1249-47a7-9472-1c8c0095fd7b None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:15.213504 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8709f98c-1249-47a7-9472-1c8c0095fd7b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:15.232920 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-8709f98c-1249-47a7-9472-1c8c0095fd7b None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:23:15.234119 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-8709f98c-1249-47a7-9472-1c8c0095fd7b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:23:15.242578 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1412/2854] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:23:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:23:15.246636 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cc62f13-c28d-4de2-a1c4-080211acc83d None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:15.256960 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cc62f13-c28d-4de2-a1c4-080211acc83d None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:15.263143 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-244f90e1-014f-4350-8af6-458a684db975 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:15.263747 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-244f90e1-014f-4350-8af6-458a684db975 admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:15.264055 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-244f90e1-014f-4350-8af6-458a684db975 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:15.264288 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-244f90e1-014f-4350-8af6-458a684db975 admin admin] PATH_INFO: `/v3/domains` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:15.265065 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-244f90e1-014f-4350-8af6-458a684db975 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:23:15.265368 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-244f90e1-014f-4350-8af6-458a684db975 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:15.267090 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-244f90e1-014f-4350-8af6-458a684db975 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:15.279110 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1443/2855] 10.222.0.77 () {58 vars in 1153 bytes} [Tue Aug 30 05:23:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:23:15.283450 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8709f98c-1249-47a7-9472-1c8c0095fd7b None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:15.292602 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8709f98c-1249-47a7-9472-1c8c0095fd7b None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:15.297168 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4daf47ea-da0a-493d-a68d-01be074019fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:15.297749 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4daf47ea-da0a-493d-a68d-01be074019fc admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:15.298026 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4daf47ea-da0a-493d-a68d-01be074019fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:15.298295 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4daf47ea-da0a-493d-a68d-01be074019fc admin admin] PATH_INFO: `/v3/projects` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:15.299461 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4daf47ea-da0a-493d-a68d-01be074019fc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:15.301347 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4daf47ea-da0a-493d-a68d-01be074019fc admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:15.327772 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1413/2856] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:23:15 2022] POST /identity/v3/projects => generated 355 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:23:15.331945 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-244f90e1-014f-4350-8af6-458a684db975 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:15.342276 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-244f90e1-014f-4350-8af6-458a684db975 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:15.397799 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d9fe937-a15c-432e-834d-8b8dbfa3f08a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:15.398396 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d9fe937-a15c-432e-834d-8b8dbfa3f08a admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:15.398675 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d9fe937-a15c-432e-834d-8b8dbfa3f08a admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:15.398978 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d9fe937-a15c-432e-834d-8b8dbfa3f08a admin admin] PATH_INFO: `/v3/users` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:15.399607 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d9fe937-a15c-432e-834d-8b8dbfa3f08a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:15.401517 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d9fe937-a15c-432e-834d-8b8dbfa3f08a admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:15.422353 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1444/2857] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:23:15 2022] POST /identity/v3/users => generated 278 bytes in 91 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 05:23:15.427081 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4daf47ea-da0a-493d-a68d-01be074019fc admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:15.437412 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4daf47ea-da0a-493d-a68d-01be074019fc admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:15.442584 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38c97996-1b02-4877-8c87-1c2dc19d3822 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:15.443209 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38c97996-1b02-4877-8c87-1c2dc19d3822 admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:15.443468 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38c97996-1b02-4877-8c87-1c2dc19d3822 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:15.443734 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38c97996-1b02-4877-8c87-1c2dc19d3822 admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:15.444418 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38c97996-1b02-4877-8c87-1c2dc19d3822 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:15.445915 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38c97996-1b02-4877-8c87-1c2dc19d3822 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:15.455756 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1414/2858] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:23:15 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:23:15.459824 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d9fe937-a15c-432e-834d-8b8dbfa3f08a admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:15.470063 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d9fe937-a15c-432e-834d-8b8dbfa3f08a admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:15.474782 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd6c03d1-49d9-4235-b403-2a4604aca134 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:15.475398 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd6c03d1-49d9-4235-b403-2a4604aca134 admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:15.475683 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd6c03d1-49d9-4235-b403-2a4604aca134 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:15.475954 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd6c03d1-49d9-4235-b403-2a4604aca134 admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:15.476590 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd6c03d1-49d9-4235-b403-2a4604aca134 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:15.478170 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd6c03d1-49d9-4235-b403-2a4604aca134 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:15.490051 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1445/2859] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:23:15 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:23:15.494427 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38c97996-1b02-4877-8c87-1c2dc19d3822 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:15.504937 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38c97996-1b02-4877-8c87-1c2dc19d3822 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:15.509564 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7721d17e-93b5-40fc-8ac7-7853cf2ca117 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:15.510197 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7721d17e-93b5-40fc-8ac7-7853cf2ca117 admin admin] REQUEST_METHOD: `PUT` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:15.510472 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7721d17e-93b5-40fc-8ac7-7853cf2ca117 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:15.510740 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7721d17e-93b5-40fc-8ac7-7853cf2ca117 admin admin] PATH_INFO: `/v3/projects/75c26c6db5f541bcbc66f560cf87492b/users/7e84f7597a174bc69cec4185f713c498/roles/6f03d86980c34c36b44a8d6764ff4fda` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:15.525315 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7721d17e-93b5-40fc-8ac7-7853cf2ca117 admin admin] RBAC: Authorizing `identity:create_grant(project_id=75c26c6db5f541bcbc66f560cf87492b, user_id=7e84f7597a174bc69cec4185f713c498, role_id=6f03d86980c34c36b44a8d6764ff4fda)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:15.527993 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7721d17e-93b5-40fc-8ac7-7853cf2ca117 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:15.540538 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1415/2860] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:23:15 2022] PUT /identity/v3/projects/75c26c6db5f541bcbc66f560cf87492b/users/7e84f7597a174bc69cec4185f713c498/roles/6f03d86980c34c36b44a8d6764ff4fda => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:23:15.551001 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d4a207-a020-4d12-9df1-f9ed2bc6da99 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:15.551287 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d4a207-a020-4d12-9df1-f9ed2bc6da99 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:15.551595 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d4a207-a020-4d12-9df1-f9ed2bc6da99 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:15.579072 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-d4d4a207-a020-4d12-9df1-f9ed2bc6da99 None None] MFA Rules not processed for user `7e84f7597a174bc69cec4185f713c498`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:23:15.580301 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-d4d4a207-a020-4d12-9df1-f9ed2bc6da99 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:23:15.643483 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1446/2861] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:23:15 2022] POST /identity/v3/auth/tokens => generated 3232 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:23:15.647834 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7721d17e-93b5-40fc-8ac7-7853cf2ca117 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:15.691742 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:23:15.710194 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7721d17e-93b5-40fc-8ac7-7853cf2ca117 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:15.715048 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cffe0a83-c766-4e36-b2d7-0c17939c39e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:15.715636 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cffe0a83-c766-4e36-b2d7-0c17939c39e9 admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:15.715905 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cffe0a83-c766-4e36-b2d7-0c17939c39e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:15.716176 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cffe0a83-c766-4e36-b2d7-0c17939c39e9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:15.716793 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cffe0a83-c766-4e36-b2d7-0c17939c39e9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:15.718665 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cffe0a83-c766-4e36-b2d7-0c17939c39e9 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:15.746739 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1416/2862] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:23:15 2022] POST /identity/v3/projects => generated 357 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 05:23:15.750759 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4d4a207-a020-4d12-9df1-f9ed2bc6da99 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:15.761320 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4d4a207-a020-4d12-9df1-f9ed2bc6da99 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:15.805668 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:23:15.818967 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73997006-1b6e-4f5d-8d45-409928a7b0c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:15.819595 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73997006-1b6e-4f5d-8d45-409928a7b0c2 admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:15.819873 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73997006-1b6e-4f5d-8d45-409928a7b0c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:15.820174 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73997006-1b6e-4f5d-8d45-409928a7b0c2 admin admin] PATH_INFO: `/v3/users` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:15.820826 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73997006-1b6e-4f5d-8d45-409928a7b0c2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:15.822767 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73997006-1b6e-4f5d-8d45-409928a7b0c2 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:15.843072 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1447/2863] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:23:15 2022] POST /identity/v3/users => generated 285 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:23:15.847483 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cffe0a83-c766-4e36-b2d7-0c17939c39e9 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:15.857484 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cffe0a83-c766-4e36-b2d7-0c17939c39e9 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:15.862541 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00fff86d-425c-48fd-a33a-2d3dd61ee53b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:15.863148 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00fff86d-425c-48fd-a33a-2d3dd61ee53b admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:15.863419 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00fff86d-425c-48fd-a33a-2d3dd61ee53b admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:15.863683 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00fff86d-425c-48fd-a33a-2d3dd61ee53b admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:15.864302 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00fff86d-425c-48fd-a33a-2d3dd61ee53b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:15.865860 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00fff86d-425c-48fd-a33a-2d3dd61ee53b admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:15.875436 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1417/2864] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:23:15 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:23:15.879239 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73997006-1b6e-4f5d-8d45-409928a7b0c2 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:15.889652 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73997006-1b6e-4f5d-8d45-409928a7b0c2 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:15.894312 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55ce2628-0fda-485c-a294-f0c1463c5a8f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:15.894992 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ce2628-0fda-485c-a294-f0c1463c5a8f admin admin] REQUEST_METHOD: `PUT` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:15.895273 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ce2628-0fda-485c-a294-f0c1463c5a8f admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:15.895546 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ce2628-0fda-485c-a294-f0c1463c5a8f admin admin] PATH_INFO: `/v3/projects/23d7a4a01836477ba97217fb46456c45/users/d85bf0caaa9e4ac081a7c16fbdb6c519/roles/9d9d5d2c270146488249b7ed943280db` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:15.910549 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55ce2628-0fda-485c-a294-f0c1463c5a8f admin admin] RBAC: Authorizing `identity:create_grant(project_id=23d7a4a01836477ba97217fb46456c45, user_id=d85bf0caaa9e4ac081a7c16fbdb6c519, role_id=9d9d5d2c270146488249b7ed943280db)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:15.913225 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55ce2628-0fda-485c-a294-f0c1463c5a8f admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:15.926584 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1448/2865] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:23:15 2022] PUT /identity/v3/projects/23d7a4a01836477ba97217fb46456c45/users/d85bf0caaa9e4ac081a7c16fbdb6c519/roles/9d9d5d2c270146488249b7ed943280db => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 05:23:15.930371 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00fff86d-425c-48fd-a33a-2d3dd61ee53b admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:15.940033 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00fff86d-425c-48fd-a33a-2d3dd61ee53b admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:15.994617 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-438c8e0f-5a6a-4d2e-a58a-da131010c818 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:15.995278 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438c8e0f-5a6a-4d2e-a58a-da131010c818 admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:15.995479 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438c8e0f-5a6a-4d2e-a58a-da131010c818 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:15.995743 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438c8e0f-5a6a-4d2e-a58a-da131010c818 admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:15.996376 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-438c8e0f-5a6a-4d2e-a58a-da131010c818 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:15.997962 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-438c8e0f-5a6a-4d2e-a58a-da131010c818 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:16.007309 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1418/2866] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:23:15 2022] GET /identity/v3/roles => generated 1420 bytes in 77 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:23:16.011441 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55ce2628-0fda-485c-a294-f0c1463c5a8f admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:16.022087 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55ce2628-0fda-485c-a294-f0c1463c5a8f admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:16.026700 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d2e0ea1-0e7e-4579-96d5-7304f3e65f4c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:16.027376 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d2e0ea1-0e7e-4579-96d5-7304f3e65f4c admin admin] REQUEST_METHOD: `PUT` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:16.027654 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d2e0ea1-0e7e-4579-96d5-7304f3e65f4c admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:16.027927 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d2e0ea1-0e7e-4579-96d5-7304f3e65f4c admin admin] PATH_INFO: `/v3/projects/23d7a4a01836477ba97217fb46456c45/users/d85bf0caaa9e4ac081a7c16fbdb6c519/roles/9d9d5d2c270146488249b7ed943280db` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:16.029835 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d2e0ea1-0e7e-4579-96d5-7304f3e65f4c admin admin] RBAC: Authorizing `identity:create_grant(project_id=23d7a4a01836477ba97217fb46456c45, user_id=d85bf0caaa9e4ac081a7c16fbdb6c519, role_id=9d9d5d2c270146488249b7ed943280db)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:16.032496 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d2e0ea1-0e7e-4579-96d5-7304f3e65f4c admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:16.043028 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1449/2867] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:23:16 2022] PUT /identity/v3/projects/23d7a4a01836477ba97217fb46456c45/users/d85bf0caaa9e4ac081a7c16fbdb6c519/roles/9d9d5d2c270146488249b7ed943280db => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:23:16.053946 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d2b887b-8aa9-436d-8102-5f8e4b25beab None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:16.054230 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d2b887b-8aa9-436d-8102-5f8e4b25beab None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:16.054495 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d2b887b-8aa9-436d-8102-5f8e4b25beab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:16.081537 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-0d2b887b-8aa9-436d-8102-5f8e4b25beab None None] MFA Rules not processed for user `d85bf0caaa9e4ac081a7c16fbdb6c519`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:23:16.082722 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-0d2b887b-8aa9-436d-8102-5f8e4b25beab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:23:16.149629 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1419/2868] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:23:16 2022] POST /identity/v3/auth/tokens => generated 3301 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:23:16.161282 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d2e0ea1-0e7e-4579-96d5-7304f3e65f4c admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:16.163001 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-0d2e0ea1-0e7e-4579-96d5-7304f3e65f4c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:23:16.219528 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d2e0ea1-0e7e-4579-96d5-7304f3e65f4c admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:16.223785 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45441981-6de1-4a7c-9a11-35bab76abb72 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '019a79b2affe478b8a30d1c538d0415c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:16.224368 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45441981-6de1-4a7c-9a11-35bab76abb72 service cinder] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:16.224642 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45441981-6de1-4a7c-9a11-35bab76abb72 service cinder] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:16.224908 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45441981-6de1-4a7c-9a11-35bab76abb72 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:16.233668 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45441981-6de1-4a7c-9a11-35bab76abb72 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:16.235106 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45441981-6de1-4a7c-9a11-35bab76abb72 service cinder] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:16.244347 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1450/2869] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:23:16 2022] GET /identity/v3/auth/tokens => generated 3301 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:23:16.321114 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d2b887b-8aa9-436d-8102-5f8e4b25beab None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:16.331578 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d2b887b-8aa9-436d-8102-5f8e4b25beab None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:16.335298 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d06c0a79-c7e6-42d8-97cc-1bb926506514 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '019a79b2affe478b8a30d1c538d0415c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:16.335962 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d06c0a79-c7e6-42d8-97cc-1bb926506514 service cinder] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:16.336248 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d06c0a79-c7e6-42d8-97cc-1bb926506514 service cinder] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:16.336521 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d06c0a79-c7e6-42d8-97cc-1bb926506514 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:16.368973 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:23:16.392482 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d06c0a79-c7e6-42d8-97cc-1bb926506514 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:16.394431 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d06c0a79-c7e6-42d8-97cc-1bb926506514 service cinder] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:16.403170 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1420/2870] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:23:16 2022] GET /identity/v3/auth/tokens => generated 3232 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:23:26.240228 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=49,keystone:INSERT=5 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:23:26.399171 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=11 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:23:31.054059 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45441981-6de1-4a7c-9a11-35bab76abb72 service cinder] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:31.114993 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45441981-6de1-4a7c-9a11-35bab76abb72 service cinder] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:31.119521 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7658946-8eff-4e61-95b0-b2dc879bd521 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:31.120122 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7658946-8eff-4e61-95b0-b2dc879bd521 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:31.120395 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7658946-8eff-4e61-95b0-b2dc879bd521 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:31.120660 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7658946-8eff-4e61-95b0-b2dc879bd521 admin admin] PATH_INFO: `/v3/users/7e84f7597a174bc69cec4185f713c498` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:31.121741 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7658946-8eff-4e61-95b0-b2dc879bd521 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7e84f7597a174bc69cec4185f713c498)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:31.123653 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7658946-8eff-4e61-95b0-b2dc879bd521 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:31.168630 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-b7658946-8eff-4e61-95b0-b2dc879bd521 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7e84f7597a174bc69cec4185f713c498'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:23:31.175746 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-b7658946-8eff-4e61-95b0-b2dc879bd521 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7e84f7597a174bc69cec4185f713c498'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:23:31.187604 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-b7658946-8eff-4e61-95b0-b2dc879bd521 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7e84f7597a174bc69cec4185f713c498'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:23:31.194768 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-b7658946-8eff-4e61-95b0-b2dc879bd521 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7e84f7597a174bc69cec4185f713c498'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:23:31.195436 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-b7658946-8eff-4e61-95b0-b2dc879bd521 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7e84f7597a174bc69cec4185f713c498'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:23:31.201062 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1451/2871] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:23:31 2022] DELETE /identity/v3/users/7e84f7597a174bc69cec4185f713c498 => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:23:31.205618 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9867015f-b0a8-4bae-a8cb-59a5ade8c051 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:31.206105 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9867015f-b0a8-4bae-a8cb-59a5ade8c051 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:31.206207 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9867015f-b0a8-4bae-a8cb-59a5ade8c051 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:31.225754 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-9867015f-b0a8-4bae-a8cb-59a5ade8c051 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:23:31.227025 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-9867015f-b0a8-4bae-a8cb-59a5ade8c051 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:23:31.283108 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1421/2872] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:23:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:23:31.292807 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7658946-8eff-4e61-95b0-b2dc879bd521 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:31.294370 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-b7658946-8eff-4e61-95b0-b2dc879bd521 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:23:31.351141 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7658946-8eff-4e61-95b0-b2dc879bd521 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:31.354803 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f5f338b-a727-4a3d-9f82-cc99d662a847 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:31.355413 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f5f338b-a727-4a3d-9f82-cc99d662a847 service neutron] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:31.355688 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f5f338b-a727-4a3d-9f82-cc99d662a847 service neutron] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:31.355959 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f5f338b-a727-4a3d-9f82-cc99d662a847 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:31.364900 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f5f338b-a727-4a3d-9f82-cc99d662a847 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:31.366353 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f5f338b-a727-4a3d-9f82-cc99d662a847 service neutron] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:31.375773 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1452/2873] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:23:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:23:31.637020 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9867015f-b0a8-4bae-a8cb-59a5ade8c051 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:31.638428 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-9867015f-b0a8-4bae-a8cb-59a5ade8c051 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:23:31.650833 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9867015f-b0a8-4bae-a8cb-59a5ade8c051 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:31.655356 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5637988f-f88b-4497-83ba-e9782148fb52 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:31.655973 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5637988f-f88b-4497-83ba-e9782148fb52 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:31.656246 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5637988f-f88b-4497-83ba-e9782148fb52 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:31.656514 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5637988f-f88b-4497-83ba-e9782148fb52 admin admin] PATH_INFO: `/v3/projects/75c26c6db5f541bcbc66f560cf87492b` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:31.657638 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5637988f-f88b-4497-83ba-e9782148fb52 admin admin] RBAC: Authorizing `identity:delete_project(project_id=75c26c6db5f541bcbc66f560cf87492b)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:31.659560 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5637988f-f88b-4497-83ba-e9782148fb52 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:31.706007 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-5637988f-f88b-4497-83ba-e9782148fb52 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '75c26c6db5f541bcbc66f560cf87492b', 'request_id': 'req-5637988f-f88b-4497-83ba-e9782148fb52'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:23:31.713223 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-5637988f-f88b-4497-83ba-e9782148fb52 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '75c26c6db5f541bcbc66f560cf87492b', 'request_id': 'req-5637988f-f88b-4497-83ba-e9782148fb52'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:23:31.719495 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-5637988f-f88b-4497-83ba-e9782148fb52 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '75c26c6db5f541bcbc66f560cf87492b', 'request_id': 'req-5637988f-f88b-4497-83ba-e9782148fb52'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:23:31.729564 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-5637988f-f88b-4497-83ba-e9782148fb52 admin admin] The token cache is being invalidate because project 75c26c6db5f541bcbc66f560cf87492b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81624) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:23:31.729847 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-5637988f-f88b-4497-83ba-e9782148fb52 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:23:31.730472 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-5637988f-f88b-4497-83ba-e9782148fb52 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:23:31.731936 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1422/2874] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:23:31 2022] DELETE /identity/v3/projects/75c26c6db5f541bcbc66f560cf87492b => generated 0 bytes in 95 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:23:31.735920 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f5f338b-a727-4a3d-9f82-cc99d662a847 service neutron] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:31.746014 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f5f338b-a727-4a3d-9f82-cc99d662a847 service neutron] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:31.764576 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:23:31.803857 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-721caca9-ebd4-44e6-a927-a3dc072fac81 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:31.804591 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-721caca9-ebd4-44e6-a927-a3dc072fac81 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:31.804875 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-721caca9-ebd4-44e6-a927-a3dc072fac81 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:31.805151 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-721caca9-ebd4-44e6-a927-a3dc072fac81 admin admin] PATH_INFO: `/v3/users/d85bf0caaa9e4ac081a7c16fbdb6c519` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:31.806269 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-721caca9-ebd4-44e6-a927-a3dc072fac81 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d85bf0caaa9e4ac081a7c16fbdb6c519)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:31.808209 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-721caca9-ebd4-44e6-a927-a3dc072fac81 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:31.853045 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-721caca9-ebd4-44e6-a927-a3dc072fac81 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd85bf0caaa9e4ac081a7c16fbdb6c519'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:23:31.859677 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-721caca9-ebd4-44e6-a927-a3dc072fac81 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd85bf0caaa9e4ac081a7c16fbdb6c519'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:23:31.872072 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-721caca9-ebd4-44e6-a927-a3dc072fac81 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd85bf0caaa9e4ac081a7c16fbdb6c519'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:23:31.878889 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-721caca9-ebd4-44e6-a927-a3dc072fac81 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd85bf0caaa9e4ac081a7c16fbdb6c519'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:23:31.879527 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-721caca9-ebd4-44e6-a927-a3dc072fac81 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd85bf0caaa9e4ac081a7c16fbdb6c519'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:23:31.886074 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1453/2875] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:23:31 2022] DELETE /identity/v3/users/d85bf0caaa9e4ac081a7c16fbdb6c519 => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:23:32.155958 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5637988f-f88b-4497-83ba-e9782148fb52 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:32.157397 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-5637988f-f88b-4497-83ba-e9782148fb52 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:23:32.171981 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5637988f-f88b-4497-83ba-e9782148fb52 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:32.229497 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a007b68-b82c-46f9-aecc-4cc131c2b295 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:32.230124 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a007b68-b82c-46f9-aecc-4cc131c2b295 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:32.230385 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a007b68-b82c-46f9-aecc-4cc131c2b295 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:32.230651 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a007b68-b82c-46f9-aecc-4cc131c2b295 admin admin] PATH_INFO: `/v3/projects/23d7a4a01836477ba97217fb46456c45` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:32.231805 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a007b68-b82c-46f9-aecc-4cc131c2b295 admin admin] RBAC: Authorizing `identity:delete_project(project_id=23d7a4a01836477ba97217fb46456c45)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:32.233714 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a007b68-b82c-46f9-aecc-4cc131c2b295 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:32.263302 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:23:32.280114 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-1a007b68-b82c-46f9-aecc-4cc131c2b295 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '23d7a4a01836477ba97217fb46456c45', 'request_id': 'req-1a007b68-b82c-46f9-aecc-4cc131c2b295'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:23:32.287634 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-1a007b68-b82c-46f9-aecc-4cc131c2b295 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '23d7a4a01836477ba97217fb46456c45', 'request_id': 'req-1a007b68-b82c-46f9-aecc-4cc131c2b295'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:23:32.293951 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-1a007b68-b82c-46f9-aecc-4cc131c2b295 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '23d7a4a01836477ba97217fb46456c45', 'request_id': 'req-1a007b68-b82c-46f9-aecc-4cc131c2b295'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:23:32.304529 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-1a007b68-b82c-46f9-aecc-4cc131c2b295 admin admin] The token cache is being invalidate because project 23d7a4a01836477ba97217fb46456c45 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81624) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:23:32.304815 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-1a007b68-b82c-46f9-aecc-4cc131c2b295 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:23:32.305442 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-1a007b68-b82c-46f9-aecc-4cc131c2b295 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:23:32.306906 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1423/2876] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:23:32 2022] DELETE /identity/v3/projects/23d7a4a01836477ba97217fb46456c45 => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:23:32.311750 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d7a3f9f-e31c-4cb1-8ef6-400cd0260805 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:32.312248 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d7a3f9f-e31c-4cb1-8ef6-400cd0260805 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:32.312360 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d7a3f9f-e31c-4cb1-8ef6-400cd0260805 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:32.332383 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-3d7a3f9f-e31c-4cb1-8ef6-400cd0260805 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:23:32.333593 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-3d7a3f9f-e31c-4cb1-8ef6-400cd0260805 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:23:32.390251 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1454/2877] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:23:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:23:32.398468 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45fe6ddd-b7f6-49b8-ace6-f185452ee52f None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:32.398872 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45fe6ddd-b7f6-49b8-ace6-f185452ee52f None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:32.399180 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45fe6ddd-b7f6-49b8-ace6-f185452ee52f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:32.418980 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-45fe6ddd-b7f6-49b8-ace6-f185452ee52f None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:23:32.420165 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-45fe6ddd-b7f6-49b8-ace6-f185452ee52f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:23:32.427794 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1424/2878] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:23:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:23:32.431862 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d7a3f9f-e31c-4cb1-8ef6-400cd0260805 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:32.441382 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d7a3f9f-e31c-4cb1-8ef6-400cd0260805 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:32.445873 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77708040-006e-4cb5-a6f8-f8e07bbe06d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:32.446451 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77708040-006e-4cb5-a6f8-f8e07bbe06d6 admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:32.446723 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77708040-006e-4cb5-a6f8-f8e07bbe06d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:32.447019 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77708040-006e-4cb5-a6f8-f8e07bbe06d6 admin admin] PATH_INFO: `/v3/domains` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:32.447647 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77708040-006e-4cb5-a6f8-f8e07bbe06d6 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:23:32.447941 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77708040-006e-4cb5-a6f8-f8e07bbe06d6 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:32.449510 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77708040-006e-4cb5-a6f8-f8e07bbe06d6 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:32.461633 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1455/2879] 10.222.0.77 () {58 vars in 1153 bytes} [Tue Aug 30 05:23:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:23:32.466170 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45fe6ddd-b7f6-49b8-ace6-f185452ee52f None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:32.475955 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45fe6ddd-b7f6-49b8-ace6-f185452ee52f None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:32.480543 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43e1fe95-2ccc-4560-ab09-b43ebfe7acde admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:32.481129 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43e1fe95-2ccc-4560-ab09-b43ebfe7acde admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:32.481404 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43e1fe95-2ccc-4560-ab09-b43ebfe7acde admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:32.481671 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43e1fe95-2ccc-4560-ab09-b43ebfe7acde admin admin] PATH_INFO: `/v3/projects` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:32.482287 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43e1fe95-2ccc-4560-ab09-b43ebfe7acde admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:32.484213 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43e1fe95-2ccc-4560-ab09-b43ebfe7acde admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:32.512177 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1425/2880] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:23:32 2022] POST /identity/v3/projects => generated 371 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:23:32.516591 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77708040-006e-4cb5-a6f8-f8e07bbe06d6 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:32.527165 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77708040-006e-4cb5-a6f8-f8e07bbe06d6 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:32.554011 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:23:32.586021 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10c9efdf-8e5c-4229-80a4-741abb090d6e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:32.586618 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10c9efdf-8e5c-4229-80a4-741abb090d6e admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:32.586933 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10c9efdf-8e5c-4229-80a4-741abb090d6e admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:32.587200 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10c9efdf-8e5c-4229-80a4-741abb090d6e admin admin] PATH_INFO: `/v3/users` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:32.587832 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10c9efdf-8e5c-4229-80a4-741abb090d6e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:32.589996 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10c9efdf-8e5c-4229-80a4-741abb090d6e admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:32.610368 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1456/2881] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:23:32 2022] POST /identity/v3/users => generated 286 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 05:23:32.614487 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43e1fe95-2ccc-4560-ab09-b43ebfe7acde admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:32.624690 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43e1fe95-2ccc-4560-ab09-b43ebfe7acde admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:32.629256 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22e39737-2823-4743-8937-182d68ba152d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:32.629825 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22e39737-2823-4743-8937-182d68ba152d admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:32.630100 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22e39737-2823-4743-8937-182d68ba152d admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:32.630366 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22e39737-2823-4743-8937-182d68ba152d admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:32.631022 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22e39737-2823-4743-8937-182d68ba152d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:32.632613 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22e39737-2823-4743-8937-182d68ba152d admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:32.642299 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1426/2882] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:23:32 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:23:32.646412 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10c9efdf-8e5c-4229-80a4-741abb090d6e admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:32.656052 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10c9efdf-8e5c-4229-80a4-741abb090d6e admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:32.660575 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1292ae8a-7203-40eb-8c17-80e620af540c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:32.661147 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1292ae8a-7203-40eb-8c17-80e620af540c admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:32.662036 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1292ae8a-7203-40eb-8c17-80e620af540c admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:32.662302 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1292ae8a-7203-40eb-8c17-80e620af540c admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:32.662965 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1292ae8a-7203-40eb-8c17-80e620af540c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:32.664519 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1292ae8a-7203-40eb-8c17-80e620af540c admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:32.674574 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1457/2883] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:23:32 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:23:32.678545 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22e39737-2823-4743-8937-182d68ba152d admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:32.688251 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22e39737-2823-4743-8937-182d68ba152d admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:32.692784 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc5e0cac-ee9c-48e0-945e-4aa6f63974e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:32.693406 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc5e0cac-ee9c-48e0-945e-4aa6f63974e9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:32.693682 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc5e0cac-ee9c-48e0-945e-4aa6f63974e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:32.693951 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc5e0cac-ee9c-48e0-945e-4aa6f63974e9 admin admin] PATH_INFO: `/v3/projects/d6fbc82a3ad74a34ac3ea6fdd2741d24/users/6051bd5c05974d3cb3c5ad08d410a9a1/roles/6f03d86980c34c36b44a8d6764ff4fda` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:32.708665 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc5e0cac-ee9c-48e0-945e-4aa6f63974e9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d6fbc82a3ad74a34ac3ea6fdd2741d24, user_id=6051bd5c05974d3cb3c5ad08d410a9a1, role_id=6f03d86980c34c36b44a8d6764ff4fda)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:32.711340 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc5e0cac-ee9c-48e0-945e-4aa6f63974e9 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:32.723895 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1427/2884] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:23:32 2022] PUT /identity/v3/projects/d6fbc82a3ad74a34ac3ea6fdd2741d24/users/6051bd5c05974d3cb3c5ad08d410a9a1/roles/6f03d86980c34c36b44a8d6764ff4fda => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:23:32.734677 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc880d2-89e6-4374-a490-7460776843ad None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:32.735001 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc880d2-89e6-4374-a490-7460776843ad None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:32.735272 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc880d2-89e6-4374-a490-7460776843ad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:32.762373 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-6cc880d2-89e6-4374-a490-7460776843ad None None] MFA Rules not processed for user `6051bd5c05974d3cb3c5ad08d410a9a1`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:23:32.763616 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-6cc880d2-89e6-4374-a490-7460776843ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:23:32.826136 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1458/2885] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:23:32 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:23:32.837764 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc5e0cac-ee9c-48e0-945e-4aa6f63974e9 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:32.839742 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-bc5e0cac-ee9c-48e0-945e-4aa6f63974e9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:23:32.895074 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc5e0cac-ee9c-48e0-945e-4aa6f63974e9 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:32.899145 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79fdff32-2fe3-4f78-89b5-544e61ef495b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '019a79b2affe478b8a30d1c538d0415c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:32.899735 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79fdff32-2fe3-4f78-89b5-544e61ef495b service cinder] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:32.900013 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79fdff32-2fe3-4f78-89b5-544e61ef495b service cinder] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:32.900281 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79fdff32-2fe3-4f78-89b5-544e61ef495b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:32.908353 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79fdff32-2fe3-4f78-89b5-544e61ef495b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:32.909777 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79fdff32-2fe3-4f78-89b5-544e61ef495b service cinder] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:32.918396 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1428/2886] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:23:32 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 81 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:23:32.939301 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cc880d2-89e6-4374-a490-7460776843ad None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:33.000599 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cc880d2-89e6-4374-a490-7460776843ad None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:33.006553 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6633135a-751d-4de6-8673-0ea2fdcfcb09 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:33.007390 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6633135a-751d-4de6-8673-0ea2fdcfcb09 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:33.007667 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6633135a-751d-4de6-8673-0ea2fdcfcb09 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:33.007937 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6633135a-751d-4de6-8673-0ea2fdcfcb09 admin admin] PATH_INFO: `/v3/users/6051bd5c05974d3cb3c5ad08d410a9a1` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:33.009247 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6633135a-751d-4de6-8673-0ea2fdcfcb09 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6051bd5c05974d3cb3c5ad08d410a9a1)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:33.011501 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6633135a-751d-4de6-8673-0ea2fdcfcb09 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:33.051479 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=92,keystone:INSERT=3,keystone:DELETE=5 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:23:33.057932 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-6633135a-751d-4de6-8673-0ea2fdcfcb09 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6051bd5c05974d3cb3c5ad08d410a9a1'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:23:33.065326 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-6633135a-751d-4de6-8673-0ea2fdcfcb09 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6051bd5c05974d3cb3c5ad08d410a9a1'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:23:33.077412 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-6633135a-751d-4de6-8673-0ea2fdcfcb09 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6051bd5c05974d3cb3c5ad08d410a9a1'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:23:33.084768 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-6633135a-751d-4de6-8673-0ea2fdcfcb09 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6051bd5c05974d3cb3c5ad08d410a9a1'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:23:33.085408 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-6633135a-751d-4de6-8673-0ea2fdcfcb09 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6051bd5c05974d3cb3c5ad08d410a9a1'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:23:33.091391 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1459/2887] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:23:32 2022] DELETE /identity/v3/users/6051bd5c05974d3cb3c5ad08d410a9a1 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:23:33.096209 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2883678-85bf-46ad-963f-84466e0fcc07 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:33.096492 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2883678-85bf-46ad-963f-84466e0fcc07 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:33.096761 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2883678-85bf-46ad-963f-84466e0fcc07 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:33.115945 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-e2883678-85bf-46ad-963f-84466e0fcc07 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:23:33.117132 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-e2883678-85bf-46ad-963f-84466e0fcc07 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:23:33.162037 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:DELETE=6,keystone:SELECT=91,keystone:INSERT=3 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:23:33.174919 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1429/2888] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:23:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:23:33.184943 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6633135a-751d-4de6-8673-0ea2fdcfcb09 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:33.186339 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-6633135a-751d-4de6-8673-0ea2fdcfcb09 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:23:33.242531 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6633135a-751d-4de6-8673-0ea2fdcfcb09 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:33.246185 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb5a008c-9f13-4b6c-bc14-f2fcac3fd3e1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:33.246772 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb5a008c-9f13-4b6c-bc14-f2fcac3fd3e1 service neutron] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:33.247082 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb5a008c-9f13-4b6c-bc14-f2fcac3fd3e1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:33.247350 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb5a008c-9f13-4b6c-bc14-f2fcac3fd3e1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:33.256837 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb5a008c-9f13-4b6c-bc14-f2fcac3fd3e1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:33.258254 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb5a008c-9f13-4b6c-bc14-f2fcac3fd3e1 service neutron] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:33.267625 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1460/2889] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:23:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:23:33.529036 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2883678-85bf-46ad-963f-84466e0fcc07 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:33.530498 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-e2883678-85bf-46ad-963f-84466e0fcc07 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:23:33.542782 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2883678-85bf-46ad-963f-84466e0fcc07 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:33.547335 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d465289-4861-4cb2-918b-f82c2378cf06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:33.547951 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d465289-4861-4cb2-918b-f82c2378cf06 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:33.548225 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d465289-4861-4cb2-918b-f82c2378cf06 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:33.548495 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d465289-4861-4cb2-918b-f82c2378cf06 admin admin] PATH_INFO: `/v3/projects/d6fbc82a3ad74a34ac3ea6fdd2741d24` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:33.549595 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d465289-4861-4cb2-918b-f82c2378cf06 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d6fbc82a3ad74a34ac3ea6fdd2741d24)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:33.551535 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d465289-4861-4cb2-918b-f82c2378cf06 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:33.598634 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-7d465289-4861-4cb2-918b-f82c2378cf06 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd6fbc82a3ad74a34ac3ea6fdd2741d24', 'request_id': 'req-7d465289-4861-4cb2-918b-f82c2378cf06'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:23:33.605947 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-7d465289-4861-4cb2-918b-f82c2378cf06 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd6fbc82a3ad74a34ac3ea6fdd2741d24', 'request_id': 'req-7d465289-4861-4cb2-918b-f82c2378cf06'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:23:33.612026 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-7d465289-4861-4cb2-918b-f82c2378cf06 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd6fbc82a3ad74a34ac3ea6fdd2741d24', 'request_id': 'req-7d465289-4861-4cb2-918b-f82c2378cf06'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:23:33.622586 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-7d465289-4861-4cb2-918b-f82c2378cf06 admin admin] The token cache is being invalidate because project d6fbc82a3ad74a34ac3ea6fdd2741d24 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81624) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:23:33.622905 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-7d465289-4861-4cb2-918b-f82c2378cf06 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:23:33.623529 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-7d465289-4861-4cb2-918b-f82c2378cf06 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:23:33.624975 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1430/2890] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:23:33 2022] DELETE /identity/v3/projects/d6fbc82a3ad74a34ac3ea6fdd2741d24 => generated 0 bytes in 96 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:23:33.630575 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dee5151-f907-4466-8eea-c70abe6c81b0 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:33.631073 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dee5151-f907-4466-8eea-c70abe6c81b0 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:33.631172 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dee5151-f907-4466-8eea-c70abe6c81b0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:33.650787 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-7dee5151-f907-4466-8eea-c70abe6c81b0 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:23:33.652113 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-7dee5151-f907-4466-8eea-c70abe6c81b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:23:33.710676 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1461/2891] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:23:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:23:33.719574 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d1ac45c-8f21-42f4-9077-5cc5c90cb937 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:33.719864 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d1ac45c-8f21-42f4-9077-5cc5c90cb937 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:33.720159 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d1ac45c-8f21-42f4-9077-5cc5c90cb937 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:33.740323 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-8d1ac45c-8f21-42f4-9077-5cc5c90cb937 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:23:33.741509 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-8d1ac45c-8f21-42f4-9077-5cc5c90cb937 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:23:33.749255 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1431/2892] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:23:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:23:33.753536 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dee5151-f907-4466-8eea-c70abe6c81b0 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:33.763165 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dee5151-f907-4466-8eea-c70abe6c81b0 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:33.767714 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3911515c-42e3-4cdf-8479-6e68e05cf8c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:33.768326 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3911515c-42e3-4cdf-8479-6e68e05cf8c7 admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:33.768610 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3911515c-42e3-4cdf-8479-6e68e05cf8c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:33.768876 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3911515c-42e3-4cdf-8479-6e68e05cf8c7 admin admin] PATH_INFO: `/v3/domains` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:33.769506 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3911515c-42e3-4cdf-8479-6e68e05cf8c7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:23:33.769803 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3911515c-42e3-4cdf-8479-6e68e05cf8c7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:33.771468 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3911515c-42e3-4cdf-8479-6e68e05cf8c7 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:33.782723 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1462/2893] 10.222.0.77 () {58 vars in 1153 bytes} [Tue Aug 30 05:23:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:23:33.787226 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d1ac45c-8f21-42f4-9077-5cc5c90cb937 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:33.796157 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d1ac45c-8f21-42f4-9077-5cc5c90cb937 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:33.800731 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af21175c-c0f2-4375-ad4e-7ff910fa0f51 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:33.801302 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af21175c-c0f2-4375-ad4e-7ff910fa0f51 admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:33.801575 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af21175c-c0f2-4375-ad4e-7ff910fa0f51 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:33.801837 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af21175c-c0f2-4375-ad4e-7ff910fa0f51 admin admin] PATH_INFO: `/v3/projects` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:33.803028 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af21175c-c0f2-4375-ad4e-7ff910fa0f51 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:33.804897 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af21175c-c0f2-4375-ad4e-7ff910fa0f51 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:33.830675 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1432/2894] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:23:33 2022] POST /identity/v3/projects => generated 385 bytes in 44 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:23:33.834785 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3911515c-42e3-4cdf-8479-6e68e05cf8c7 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:33.844880 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3911515c-42e3-4cdf-8479-6e68e05cf8c7 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:33.888770 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:DELETE=5,keystone:SELECT=94,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:23:33.903796 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbcabfcf-35e9-47aa-91b5-a607a981330d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:33.904393 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbcabfcf-35e9-47aa-91b5-a607a981330d admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:33.904674 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbcabfcf-35e9-47aa-91b5-a607a981330d admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:33.904943 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbcabfcf-35e9-47aa-91b5-a607a981330d admin admin] PATH_INFO: `/v3/users` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:33.905560 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbcabfcf-35e9-47aa-91b5-a607a981330d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:33.907610 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbcabfcf-35e9-47aa-91b5-a607a981330d admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:33.929590 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1463/2895] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:23:33 2022] POST /identity/v3/users => generated 293 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:23:33.933625 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af21175c-c0f2-4375-ad4e-7ff910fa0f51 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:33.947628 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af21175c-c0f2-4375-ad4e-7ff910fa0f51 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:33.952246 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37a6cd51-f06b-4a2e-9583-8346405e9d2b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:33.952838 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37a6cd51-f06b-4a2e-9583-8346405e9d2b admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:33.953115 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37a6cd51-f06b-4a2e-9583-8346405e9d2b admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:33.953384 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37a6cd51-f06b-4a2e-9583-8346405e9d2b admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:33.954018 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37a6cd51-f06b-4a2e-9583-8346405e9d2b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:33.955654 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37a6cd51-f06b-4a2e-9583-8346405e9d2b admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:33.969119 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1433/2896] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:23:33 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:23:33.970221 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbcabfcf-35e9-47aa-91b5-a607a981330d admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:33.981508 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbcabfcf-35e9-47aa-91b5-a607a981330d admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:33.986206 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5f24e09-ad12-449c-b48f-80455bec65bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:33.986802 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5f24e09-ad12-449c-b48f-80455bec65bf admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:33.987108 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5f24e09-ad12-449c-b48f-80455bec65bf admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:33.987376 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5f24e09-ad12-449c-b48f-80455bec65bf admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:33.988007 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5f24e09-ad12-449c-b48f-80455bec65bf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:33.989621 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5f24e09-ad12-449c-b48f-80455bec65bf admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:34.001327 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1464/2897] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:23:33 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 05:23:34.005486 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37a6cd51-f06b-4a2e-9583-8346405e9d2b admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:34.015626 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37a6cd51-f06b-4a2e-9583-8346405e9d2b admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:34.020238 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8a134c9-d5a1-487e-b440-760b94c1214c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:34.021491 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8a134c9-d5a1-487e-b440-760b94c1214c admin admin] REQUEST_METHOD: `PUT` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:34.021790 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8a134c9-d5a1-487e-b440-760b94c1214c admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:34.022047 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8a134c9-d5a1-487e-b440-760b94c1214c admin admin] PATH_INFO: `/v3/projects/bc55d335b3454cb582c04ecd166e46b3/users/56653bceeb434a51952a58e4907a7b1a/roles/6f03d86980c34c36b44a8d6764ff4fda` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:34.037056 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8a134c9-d5a1-487e-b440-760b94c1214c admin admin] RBAC: Authorizing `identity:create_grant(project_id=bc55d335b3454cb582c04ecd166e46b3, user_id=56653bceeb434a51952a58e4907a7b1a, role_id=6f03d86980c34c36b44a8d6764ff4fda)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:34.039717 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8a134c9-d5a1-487e-b440-760b94c1214c admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:34.052817 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1434/2898] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:23:34 2022] PUT /identity/v3/projects/bc55d335b3454cb582c04ecd166e46b3/users/56653bceeb434a51952a58e4907a7b1a/roles/6f03d86980c34c36b44a8d6764ff4fda => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:23:34.063171 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bcce2a9-1e16-4034-9341-9289432e67e8 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:34.063456 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bcce2a9-1e16-4034-9341-9289432e67e8 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:34.063724 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bcce2a9-1e16-4034-9341-9289432e67e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:34.091100 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-0bcce2a9-1e16-4034-9341-9289432e67e8 None None] MFA Rules not processed for user `56653bceeb434a51952a58e4907a7b1a`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:23:34.092293 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-0bcce2a9-1e16-4034-9341-9289432e67e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:23:34.154769 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1465/2899] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:23:34 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:23:34.166362 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8a134c9-d5a1-487e-b440-760b94c1214c admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:34.168056 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-f8a134c9-d5a1-487e-b440-760b94c1214c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:23:34.223123 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8a134c9-d5a1-487e-b440-760b94c1214c admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:34.227288 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c3c9c20-f82c-4458-a08d-ea96b139d093 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '019a79b2affe478b8a30d1c538d0415c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:34.227883 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c3c9c20-f82c-4458-a08d-ea96b139d093 service cinder] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:34.228161 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c3c9c20-f82c-4458-a08d-ea96b139d093 service cinder] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:34.228430 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c3c9c20-f82c-4458-a08d-ea96b139d093 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:34.236534 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c3c9c20-f82c-4458-a08d-ea96b139d093 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:34.237971 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c3c9c20-f82c-4458-a08d-ea96b139d093 service cinder] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:34.246838 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1435/2900] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:23:34 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 81 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:23:36.775922 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bcce2a9-1e16-4034-9341-9289432e67e8 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:36.837291 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bcce2a9-1e16-4034-9341-9289432e67e8 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:36.842048 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bf803b8-cdc3-4a76-83fc-6e9d21f2f4e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:36.842656 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf803b8-cdc3-4a76-83fc-6e9d21f2f4e3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:36.842976 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf803b8-cdc3-4a76-83fc-6e9d21f2f4e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:36.843250 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf803b8-cdc3-4a76-83fc-6e9d21f2f4e3 admin admin] PATH_INFO: `/v3/users/56653bceeb434a51952a58e4907a7b1a` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:36.844399 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bf803b8-cdc3-4a76-83fc-6e9d21f2f4e3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=56653bceeb434a51952a58e4907a7b1a)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:36.846303 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bf803b8-cdc3-4a76-83fc-6e9d21f2f4e3 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:36.891737 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-0bf803b8-cdc3-4a76-83fc-6e9d21f2f4e3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '56653bceeb434a51952a58e4907a7b1a'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:23:36.899741 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-0bf803b8-cdc3-4a76-83fc-6e9d21f2f4e3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '56653bceeb434a51952a58e4907a7b1a'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:23:36.901838 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=88,keystone:INSERT=4,keystone:DELETE=8 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:23:36.914322 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-0bf803b8-cdc3-4a76-83fc-6e9d21f2f4e3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '56653bceeb434a51952a58e4907a7b1a'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:23:36.921067 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-0bf803b8-cdc3-4a76-83fc-6e9d21f2f4e3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '56653bceeb434a51952a58e4907a7b1a'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:23:36.921780 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-0bf803b8-cdc3-4a76-83fc-6e9d21f2f4e3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '56653bceeb434a51952a58e4907a7b1a'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:23:36.927808 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1466/2901] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:23:36 2022] DELETE /identity/v3/users/56653bceeb434a51952a58e4907a7b1a => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:23:36.932280 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d17948bf-c6cb-43ae-9e3d-004c00f5cd6f None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:36.932746 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d17948bf-c6cb-43ae-9e3d-004c00f5cd6f None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:36.932845 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d17948bf-c6cb-43ae-9e3d-004c00f5cd6f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:36.951185 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-d17948bf-c6cb-43ae-9e3d-004c00f5cd6f None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:23:36.952370 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-d17948bf-c6cb-43ae-9e3d-004c00f5cd6f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:23:36.961836 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:23:37.011602 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1436/2902] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:23:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:23:37.021237 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bf803b8-cdc3-4a76-83fc-6e9d21f2f4e3 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:37.023131 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-0bf803b8-cdc3-4a76-83fc-6e9d21f2f4e3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:23:37.079937 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bf803b8-cdc3-4a76-83fc-6e9d21f2f4e3 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:37.083530 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-249db087-1212-4287-af1c-8bdb2c7dd00a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:37.084679 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249db087-1212-4287-af1c-8bdb2c7dd00a service neutron] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:37.084952 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249db087-1212-4287-af1c-8bdb2c7dd00a service neutron] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:37.085217 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249db087-1212-4287-af1c-8bdb2c7dd00a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:37.094248 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-249db087-1212-4287-af1c-8bdb2c7dd00a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:37.095674 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-249db087-1212-4287-af1c-8bdb2c7dd00a service neutron] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:37.105421 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1467/2903] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:23:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:23:37.363617 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d17948bf-c6cb-43ae-9e3d-004c00f5cd6f None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:37.365052 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-d17948bf-c6cb-43ae-9e3d-004c00f5cd6f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:23:37.376968 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d17948bf-c6cb-43ae-9e3d-004c00f5cd6f None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:37.382090 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb964006-eb00-4f7a-86e3-e6ad32495b95 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:37.382714 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb964006-eb00-4f7a-86e3-e6ad32495b95 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:37.383025 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb964006-eb00-4f7a-86e3-e6ad32495b95 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:37.383321 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb964006-eb00-4f7a-86e3-e6ad32495b95 admin admin] PATH_INFO: `/v3/projects/bc55d335b3454cb582c04ecd166e46b3` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:37.384432 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb964006-eb00-4f7a-86e3-e6ad32495b95 admin admin] RBAC: Authorizing `identity:delete_project(project_id=bc55d335b3454cb582c04ecd166e46b3)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:37.386332 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb964006-eb00-4f7a-86e3-e6ad32495b95 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:37.432660 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-fb964006-eb00-4f7a-86e3-e6ad32495b95 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bc55d335b3454cb582c04ecd166e46b3', 'request_id': 'req-fb964006-eb00-4f7a-86e3-e6ad32495b95'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:23:37.442258 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-fb964006-eb00-4f7a-86e3-e6ad32495b95 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bc55d335b3454cb582c04ecd166e46b3', 'request_id': 'req-fb964006-eb00-4f7a-86e3-e6ad32495b95'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:23:37.449089 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-fb964006-eb00-4f7a-86e3-e6ad32495b95 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bc55d335b3454cb582c04ecd166e46b3', 'request_id': 'req-fb964006-eb00-4f7a-86e3-e6ad32495b95'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:23:37.460203 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-fb964006-eb00-4f7a-86e3-e6ad32495b95 admin admin] The token cache is being invalidate because project bc55d335b3454cb582c04ecd166e46b3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81624) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:23:37.460519 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-fb964006-eb00-4f7a-86e3-e6ad32495b95 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:23:37.461172 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-fb964006-eb00-4f7a-86e3-e6ad32495b95 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:23:37.462660 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1437/2904] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:23:37 2022] DELETE /identity/v3/projects/bc55d335b3454cb582c04ecd166e46b3 => generated 0 bytes in 100 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:23:37.467936 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b4e9bc0-e455-475f-85fc-1e7eb5c9fe39 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:37.468139 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b4e9bc0-e455-475f-85fc-1e7eb5c9fe39 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:37.468407 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b4e9bc0-e455-475f-85fc-1e7eb5c9fe39 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:37.487573 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-4b4e9bc0-e455-475f-85fc-1e7eb5c9fe39 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:23:37.488880 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-4b4e9bc0-e455-475f-85fc-1e7eb5c9fe39 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:23:37.547523 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1468/2905] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:23:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:23:37.555341 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38c3f55b-6c97-4151-8b99-5b733a1c6499 None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:37.555623 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38c3f55b-6c97-4151-8b99-5b733a1c6499 None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:37.555891 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38c3f55b-6c97-4151-8b99-5b733a1c6499 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:37.575800 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-38c3f55b-6c97-4151-8b99-5b733a1c6499 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:23:37.576996 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-38c3f55b-6c97-4151-8b99-5b733a1c6499 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:23:37.585365 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1438/2906] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:23:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:23:37.589829 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b4e9bc0-e455-475f-85fc-1e7eb5c9fe39 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:37.599594 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b4e9bc0-e455-475f-85fc-1e7eb5c9fe39 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:37.604109 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2750d696-b499-4a4d-8fb3-aaa594290eff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:37.604693 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2750d696-b499-4a4d-8fb3-aaa594290eff admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:37.604969 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2750d696-b499-4a4d-8fb3-aaa594290eff admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:37.605238 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2750d696-b499-4a4d-8fb3-aaa594290eff admin admin] PATH_INFO: `/v3/domains` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:37.605892 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2750d696-b499-4a4d-8fb3-aaa594290eff admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81625) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 05:23:37.606189 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2750d696-b499-4a4d-8fb3-aaa594290eff admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:37.607791 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2750d696-b499-4a4d-8fb3-aaa594290eff admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:37.619617 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1469/2907] 10.222.0.77 () {58 vars in 1153 bytes} [Tue Aug 30 05:23:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 05:23:37.623757 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38c3f55b-6c97-4151-8b99-5b733a1c6499 None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:37.633326 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38c3f55b-6c97-4151-8b99-5b733a1c6499 None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:37.637900 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e51263fc-ae52-401d-a793-c05ae30e1acc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:37.638472 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e51263fc-ae52-401d-a793-c05ae30e1acc admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:37.638747 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e51263fc-ae52-401d-a793-c05ae30e1acc admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:37.639045 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e51263fc-ae52-401d-a793-c05ae30e1acc admin admin] PATH_INFO: `/v3/projects` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:37.639664 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e51263fc-ae52-401d-a793-c05ae30e1acc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:37.641534 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e51263fc-ae52-401d-a793-c05ae30e1acc admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:37.667244 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1439/2908] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:23:37 2022] POST /identity/v3/projects => generated 373 bytes in 44 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:23:37.671251 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2750d696-b499-4a4d-8fb3-aaa594290eff admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:37.681526 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2750d696-b499-4a4d-8fb3-aaa594290eff admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:37.736503 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d888ef8-bdf3-426a-ae43-722f40a9d5fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:37.737085 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d888ef8-bdf3-426a-ae43-722f40a9d5fd admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:37.737379 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d888ef8-bdf3-426a-ae43-722f40a9d5fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:37.737643 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d888ef8-bdf3-426a-ae43-722f40a9d5fd admin admin] PATH_INFO: `/v3/users` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:37.738246 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d888ef8-bdf3-426a-ae43-722f40a9d5fd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:37.740653 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d888ef8-bdf3-426a-ae43-722f40a9d5fd admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:37.750214 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=96,keystone:DELETE=2,keystone:INSERT=2 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:23:37.762717 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1470/2909] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:23:37 2022] POST /identity/v3/users => generated 287 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:23:37.766726 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e51263fc-ae52-401d-a793-c05ae30e1acc admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:37.776727 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e51263fc-ae52-401d-a793-c05ae30e1acc admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:37.781150 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41458a95-68e5-4276-8c22-ef1f586daa3f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:37.781727 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41458a95-68e5-4276-8c22-ef1f586daa3f admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:37.781999 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41458a95-68e5-4276-8c22-ef1f586daa3f admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:37.782263 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41458a95-68e5-4276-8c22-ef1f586daa3f admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:37.782917 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41458a95-68e5-4276-8c22-ef1f586daa3f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:37.784476 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41458a95-68e5-4276-8c22-ef1f586daa3f admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:37.794060 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1440/2910] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:23:37 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 05:23:37.797732 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d888ef8-bdf3-426a-ae43-722f40a9d5fd admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:37.807886 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d888ef8-bdf3-426a-ae43-722f40a9d5fd admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:37.812376 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b303e44-cd41-4625-bb76-fe060f520c1b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:37.812952 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b303e44-cd41-4625-bb76-fe060f520c1b admin admin] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:37.813225 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b303e44-cd41-4625-bb76-fe060f520c1b admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:37.813488 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b303e44-cd41-4625-bb76-fe060f520c1b admin admin] PATH_INFO: `/v3/roles` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:37.814106 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b303e44-cd41-4625-bb76-fe060f520c1b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:37.815736 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b303e44-cd41-4625-bb76-fe060f520c1b admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:37.826477 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1471/2911] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:23:37 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:23:37.830514 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41458a95-68e5-4276-8c22-ef1f586daa3f admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:37.840368 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41458a95-68e5-4276-8c22-ef1f586daa3f admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:37.844844 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3997a4e3-08db-4674-9fb8-e2e2c377441d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:37.845477 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3997a4e3-08db-4674-9fb8-e2e2c377441d admin admin] REQUEST_METHOD: `PUT` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:37.845756 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3997a4e3-08db-4674-9fb8-e2e2c377441d admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:37.846014 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3997a4e3-08db-4674-9fb8-e2e2c377441d admin admin] PATH_INFO: `/v3/projects/05ebb49967334f8bb6f2077fce987ee3/users/8f9b73d97e8145a39215f179b1398350/roles/6f03d86980c34c36b44a8d6764ff4fda` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:37.860386 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3997a4e3-08db-4674-9fb8-e2e2c377441d admin admin] RBAC: Authorizing `identity:create_grant(project_id=05ebb49967334f8bb6f2077fce987ee3, user_id=8f9b73d97e8145a39215f179b1398350, role_id=6f03d86980c34c36b44a8d6764ff4fda)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:37.863013 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3997a4e3-08db-4674-9fb8-e2e2c377441d admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:37.876397 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1441/2912] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:23:37 2022] PUT /identity/v3/projects/05ebb49967334f8bb6f2077fce987ee3/users/8f9b73d97e8145a39215f179b1398350/roles/6f03d86980c34c36b44a8d6764ff4fda => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 05:23:37.881791 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08d96f39-23a4-4609-99fd-854bb238e0c6 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:37.882079 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08d96f39-23a4-4609-99fd-854bb238e0c6 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:37.882380 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08d96f39-23a4-4609-99fd-854bb238e0c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:37.901192 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-08d96f39-23a4-4609-99fd-854bb238e0c6 None None] MFA Rules not processed for user `fdab92bac3f64739a0fcb5f729d40f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:23:37.902432 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-08d96f39-23a4-4609-99fd-854bb238e0c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:23:37.960149 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1472/2913] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:23:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:23:37.971412 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3997a4e3-08db-4674-9fb8-e2e2c377441d admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:37.973143 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-3997a4e3-08db-4674-9fb8-e2e2c377441d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:23:38.013015 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:23:38.031809 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3997a4e3-08db-4674-9fb8-e2e2c377441d admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:38.035420 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e117d27e-16a3-4e52-8530-41054075bb61 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:38.036013 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e117d27e-16a3-4e52-8530-41054075bb61 service neutron] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:38.036291 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e117d27e-16a3-4e52-8530-41054075bb61 service neutron] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:38.036561 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e117d27e-16a3-4e52-8530-41054075bb61 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:38.045804 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e117d27e-16a3-4e52-8530-41054075bb61 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:38.047571 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e117d27e-16a3-4e52-8530-41054075bb61 service neutron] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:38.057967 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1442/2914] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:23:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:23:41.954897 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c07ff25-f9b5-492b-a5df-cf287b1a04d4 None None] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:41.955733 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c07ff25-f9b5-492b-a5df-cf287b1a04d4 None None] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:41.956011 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c07ff25-f9b5-492b-a5df-cf287b1a04d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:41.988673 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.auth.core [None req-8c07ff25-f9b5-492b-a5df-cf287b1a04d4 None None] MFA Rules not processed for user `8f9b73d97e8145a39215f179b1398350`. Rule list: `[]` (Enabled: `True`). {{(pid=81625) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:23:41.990978 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-8c07ff25-f9b5-492b-a5df-cf287b1a04d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:23:42.061851 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1473/2915] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:23:41 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:23:42.066476 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e117d27e-16a3-4e52-8530-41054075bb61 service neutron] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:42.077101 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e117d27e-16a3-4e52-8530-41054075bb61 service neutron] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:42.081657 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06f4711b-faa2-4800-857d-f42aec6a836f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:42.082255 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f4711b-faa2-4800-857d-f42aec6a836f admin admin] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:42.082526 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f4711b-faa2-4800-857d-f42aec6a836f admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:42.082847 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f4711b-faa2-4800-857d-f42aec6a836f admin admin] PATH_INFO: `/v3/projects` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:42.083473 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06f4711b-faa2-4800-857d-f42aec6a836f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:42.085890 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06f4711b-faa2-4800-857d-f42aec6a836f admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:42.115926 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1443/2916] 10.222.0.77 () {60 vars in 1153 bytes} [Tue Aug 30 05:23:42 2022] POST /identity/v3/projects => generated 371 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 05:23:42.120177 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c07ff25-f9b5-492b-a5df-cf287b1a04d4 None None] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:42.131722 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c07ff25-f9b5-492b-a5df-cf287b1a04d4 None None] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:42.188297 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4ed9e95-2685-45ce-a70c-7aecae2b2045 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:42.188895 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4ed9e95-2685-45ce-a70c-7aecae2b2045 admin admin] REQUEST_METHOD: `POST` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:42.189177 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4ed9e95-2685-45ce-a70c-7aecae2b2045 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:42.189450 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4ed9e95-2685-45ce-a70c-7aecae2b2045 admin admin] PATH_INFO: `/v3/users` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:42.190080 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4ed9e95-2685-45ce-a70c-7aecae2b2045 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:42.192037 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4ed9e95-2685-45ce-a70c-7aecae2b2045 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:42.212693 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1474/2917] 10.222.0.77 () {60 vars in 1144 bytes} [Tue Aug 30 05:23:42 2022] POST /identity/v3/users => generated 292 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 05:23:42.217010 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06f4711b-faa2-4800-857d-f42aec6a836f admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:42.228400 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06f4711b-faa2-4800-857d-f42aec6a836f admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:42.232974 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b2a5db4-914b-4912-90f2-b485e7cb6c18 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:42.233567 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b2a5db4-914b-4912-90f2-b485e7cb6c18 admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:42.233845 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b2a5db4-914b-4912-90f2-b485e7cb6c18 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:42.234115 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b2a5db4-914b-4912-90f2-b485e7cb6c18 admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:42.234749 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b2a5db4-914b-4912-90f2-b485e7cb6c18 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:42.236477 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b2a5db4-914b-4912-90f2-b485e7cb6c18 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:42.246918 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1444/2918] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:23:42 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:23:42.250987 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4ed9e95-2685-45ce-a70c-7aecae2b2045 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:42.256096 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:INSERT=4,keystone:SELECT=96 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:23:42.262600 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4ed9e95-2685-45ce-a70c-7aecae2b2045 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:42.267325 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94a93e9d-af39-46c4-9832-2bf8d0b84dd0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:42.267963 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a93e9d-af39-46c4-9832-2bf8d0b84dd0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:42.268237 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a93e9d-af39-46c4-9832-2bf8d0b84dd0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:42.268506 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a93e9d-af39-46c4-9832-2bf8d0b84dd0 admin admin] PATH_INFO: `/v3/projects/067b159ec95a4432b30aa1d3fc3f98b4/users/86ccd1436c3d4739a3ef88f3ec94193a/roles/9d9d5d2c270146488249b7ed943280db` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:42.283515 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94a93e9d-af39-46c4-9832-2bf8d0b84dd0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=067b159ec95a4432b30aa1d3fc3f98b4, user_id=86ccd1436c3d4739a3ef88f3ec94193a, role_id=9d9d5d2c270146488249b7ed943280db)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:42.286123 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94a93e9d-af39-46c4-9832-2bf8d0b84dd0 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:42.299181 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1475/2919] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:23:42 2022] PUT /identity/v3/projects/067b159ec95a4432b30aa1d3fc3f98b4/users/86ccd1436c3d4739a3ef88f3ec94193a/roles/9d9d5d2c270146488249b7ed943280db => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 05:23:42.303310 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b2a5db4-914b-4912-90f2-b485e7cb6c18 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:42.313204 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b2a5db4-914b-4912-90f2-b485e7cb6c18 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:42.369754 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67bcbbb8-4307-4695-b264-51c2fa6cdf4f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:42.370352 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67bcbbb8-4307-4695-b264-51c2fa6cdf4f admin admin] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:42.370629 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67bcbbb8-4307-4695-b264-51c2fa6cdf4f admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:42.370937 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67bcbbb8-4307-4695-b264-51c2fa6cdf4f admin admin] PATH_INFO: `/v3/roles` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:42.371559 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67bcbbb8-4307-4695-b264-51c2fa6cdf4f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:42.373171 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67bcbbb8-4307-4695-b264-51c2fa6cdf4f admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:42.382680 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1445/2920] 10.222.0.77 () {58 vars in 1122 bytes} [Tue Aug 30 05:23:42 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 05:23:42.386840 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94a93e9d-af39-46c4-9832-2bf8d0b84dd0 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:42.397431 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94a93e9d-af39-46c4-9832-2bf8d0b84dd0 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:42.402136 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ce16c9b-d9d8-49eb-889b-92b18d241e24 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:42.402761 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce16c9b-d9d8-49eb-889b-92b18d241e24 admin admin] REQUEST_METHOD: `PUT` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:42.403071 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce16c9b-d9d8-49eb-889b-92b18d241e24 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:42.403358 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce16c9b-d9d8-49eb-889b-92b18d241e24 admin admin] PATH_INFO: `/v3/projects/067b159ec95a4432b30aa1d3fc3f98b4/users/86ccd1436c3d4739a3ef88f3ec94193a/roles/9d9d5d2c270146488249b7ed943280db` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:42.405286 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ce16c9b-d9d8-49eb-889b-92b18d241e24 admin admin] RBAC: Authorizing `identity:create_grant(project_id=067b159ec95a4432b30aa1d3fc3f98b4, user_id=86ccd1436c3d4739a3ef88f3ec94193a, role_id=9d9d5d2c270146488249b7ed943280db)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:42.407947 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ce16c9b-d9d8-49eb-889b-92b18d241e24 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:42.418579 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1476/2921] 10.222.0.77 () {60 vars in 1483 bytes} [Tue Aug 30 05:23:42 2022] PUT /identity/v3/projects/067b159ec95a4432b30aa1d3fc3f98b4/users/86ccd1436c3d4739a3ef88f3ec94193a/roles/9d9d5d2c270146488249b7ed943280db => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:23:46.731336 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5552442d-8554-4a65-a82b-116a0674dd9c None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:46.731862 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5552442d-8554-4a65-a82b-116a0674dd9c None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:46.732314 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5552442d-8554-4a65-a82b-116a0674dd9c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:46.765246 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-5552442d-8554-4a65-a82b-116a0674dd9c None None] MFA Rules not processed for user `86ccd1436c3d4739a3ef88f3ec94193a`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:23:46.766657 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-5552442d-8554-4a65-a82b-116a0674dd9c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:23:46.854865 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1446/2922] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:23:46 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:23:46.870897 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ce16c9b-d9d8-49eb-889b-92b18d241e24 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:23:46.874028 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-3ce16c9b-d9d8-49eb-889b-92b18d241e24 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:23:46.953339 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ce16c9b-d9d8-49eb-889b-92b18d241e24 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:23:46.959036 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa2f742f-eac4-4ef7-a602-d69a369285af service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '886eb5e00bbf4ec292eb962b665b37d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:23:46.959646 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa2f742f-eac4-4ef7-a602-d69a369285af service nova] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:46.959929 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa2f742f-eac4-4ef7-a602-d69a369285af service nova] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:46.960202 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa2f742f-eac4-4ef7-a602-d69a369285af service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:47.019382 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa2f742f-eac4-4ef7-a602-d69a369285af service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:23:47.021024 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa2f742f-eac4-4ef7-a602-d69a369285af service nova] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:23:47.030866 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1477/2923] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:23:46 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 161 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:23:48.012919 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a5b1985-00c5-4550-a515-78dbef8b2f4a None None] REQUEST_METHOD: `POST` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:23:48.013489 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a5b1985-00c5-4550-a515-78dbef8b2f4a None None] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:23:48.013604 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a5b1985-00c5-4550-a515-78dbef8b2f4a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:23:48.034632 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.auth.core [None req-3a5b1985-00c5-4550-a515-78dbef8b2f4a None None] MFA Rules not processed for user `8f9b73d97e8145a39215f179b1398350`. Rule list: `[]` (Enabled: `True`). {{(pid=81624) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 05:23:48.035886 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-3a5b1985-00c5-4550-a515-78dbef8b2f4a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:23:48.043711 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1447/2924] 10.222.0.77 () {58 vars in 958 bytes} [Tue Aug 30 05:23:48 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 05:23:57.026427 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=70,keystone:INSERT=2 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:23:58.028775 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=90,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:24:04.629873 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa2f742f-eac4-4ef7-a602-d69a369285af service nova] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:24:04.685010 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa2f742f-eac4-4ef7-a602-d69a369285af service nova] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:24:04.688638 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e03f39f-f44d-474f-a046-d38b08d369d2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:24:04.689232 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e03f39f-f44d-474f-a046-d38b08d369d2 service neutron] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:24:04.689513 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e03f39f-f44d-474f-a046-d38b08d369d2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:24:04.689783 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e03f39f-f44d-474f-a046-d38b08d369d2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:24:04.691529 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-0e03f39f-f44d-474f-a046-d38b08d369d2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:24:04.701086 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e03f39f-f44d-474f-a046-d38b08d369d2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:24:04.702554 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e03f39f-f44d-474f-a046-d38b08d369d2 service neutron] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:24:04.710909 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1478/2925] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:24:04 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 82 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:24:05.158674 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a5b1985-00c5-4550-a515-78dbef8b2f4a None None] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:24:05.160225 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-3a5b1985-00c5-4550-a515-78dbef8b2f4a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:24:05.230604 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a5b1985-00c5-4550-a515-78dbef8b2f4a None None] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:24:05.235644 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-703001b3-dece-4fd8-9e71-050d39e2547f service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1972959402704b89925001cd8d90cb0e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:24:05.236309 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-703001b3-dece-4fd8-9e71-050d39e2547f service placement] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:24:05.236592 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-703001b3-dece-4fd8-9e71-050d39e2547f service placement] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:24:05.236868 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-703001b3-dece-4fd8-9e71-050d39e2547f service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:24:05.238564 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-703001b3-dece-4fd8-9e71-050d39e2547f service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:24:05.250232 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-703001b3-dece-4fd8-9e71-050d39e2547f service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:24:05.251885 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-703001b3-dece-4fd8-9e71-050d39e2547f service placement] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:24:05.262066 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1448/2926] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:24:05 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:24:06.998030 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e03f39f-f44d-474f-a046-d38b08d369d2 service neutron] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:24:06.999492 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-0e03f39f-f44d-474f-a046-d38b08d369d2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:24:07.010500 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e03f39f-f44d-474f-a046-d38b08d369d2 service neutron] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:24:07.015213 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-370649cd-f3be-4649-a3a7-999871b62552 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:24:07.016008 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-370649cd-f3be-4649-a3a7-999871b62552 service neutron] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:24:07.016289 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-370649cd-f3be-4649-a3a7-999871b62552 service neutron] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:24:07.016563 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-370649cd-f3be-4649-a3a7-999871b62552 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:24:07.018371 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-370649cd-f3be-4649-a3a7-999871b62552 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:24:07.031206 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-370649cd-f3be-4649-a3a7-999871b62552 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:24:07.032582 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-370649cd-f3be-4649-a3a7-999871b62552 service neutron] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:24:07.042463 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1479/2927] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:24:06 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:24:15.257934 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=32 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:24:17.038498 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=32 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:24:18.819400 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-703001b3-dece-4fd8-9e71-050d39e2547f service placement] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:24:18.821569 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-703001b3-dece-4fd8-9e71-050d39e2547f service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:24:18.836731 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-703001b3-dece-4fd8-9e71-050d39e2547f service placement] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:24:18.842582 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-965372f0-2e8b-4c62-bf1d-6489149e0958 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '886eb5e00bbf4ec292eb962b665b37d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:24:18.843214 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-965372f0-2e8b-4c62-bf1d-6489149e0958 service nova] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:24:18.843506 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-965372f0-2e8b-4c62-bf1d-6489149e0958 service nova] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:24:18.843762 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-965372f0-2e8b-4c62-bf1d-6489149e0958 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:24:18.845459 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-965372f0-2e8b-4c62-bf1d-6489149e0958 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:24:18.857993 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-965372f0-2e8b-4c62-bf1d-6489149e0958 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:24:18.859713 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-965372f0-2e8b-4c62-bf1d-6489149e0958 service nova] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:24:18.870864 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1449/2928] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:24:18 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 56 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:24:22.139288 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-370649cd-f3be-4649-a3a7-999871b62552 service neutron] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:24:22.147791 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-370649cd-f3be-4649-a3a7-999871b62552 service neutron] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:24:22.151345 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9d466be-1bd6-459a-9359-a368af96408c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:24:22.151940 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9d466be-1bd6-459a-9359-a368af96408c service neutron] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:24:22.152199 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9d466be-1bd6-459a-9359-a368af96408c service neutron] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:24:22.152463 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9d466be-1bd6-459a-9359-a368af96408c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:24:22.161112 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9d466be-1bd6-459a-9359-a368af96408c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:24:22.162515 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9d466be-1bd6-459a-9359-a368af96408c service neutron] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:24:22.172245 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1480/2929] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:24:22 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 33 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:24:28.865802 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=6 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:24:32.168158 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=6 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:26:05.382655 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-965372f0-2e8b-4c62-bf1d-6489149e0958 service nova] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:26:05.384619 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-965372f0-2e8b-4c62-bf1d-6489149e0958 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:26:05.398184 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-965372f0-2e8b-4c62-bf1d-6489149e0958 service nova] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:26:05.403643 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e21a9751-4d5c-49db-870b-45d837ae63fb service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1972959402704b89925001cd8d90cb0e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:26:05.404251 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e21a9751-4d5c-49db-870b-45d837ae63fb service placement] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:26:05.404522 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e21a9751-4d5c-49db-870b-45d837ae63fb service placement] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:26:05.404786 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e21a9751-4d5c-49db-870b-45d837ae63fb service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:26:05.406416 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-e21a9751-4d5c-49db-870b-45d837ae63fb service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:26:05.417355 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e21a9751-4d5c-49db-870b-45d837ae63fb service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:26:05.418939 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e21a9751-4d5c-49db-870b-45d837ae63fb service placement] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:26:05.428756 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1450/2930] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:26:05 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 47 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:26:15.424533 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=6 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:26:34.442168 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9d466be-1bd6-459a-9359-a368af96408c service neutron] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:26:34.451605 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9d466be-1bd6-459a-9359-a368af96408c service neutron] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:26:34.456892 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccf08d9b-fa07-47a6-87f9-e124ac61f57e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:26:34.456892 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccf08d9b-fa07-47a6-87f9-e124ac61f57e service neutron] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:26:34.456892 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccf08d9b-fa07-47a6-87f9-e124ac61f57e service neutron] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:26:34.456892 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccf08d9b-fa07-47a6-87f9-e124ac61f57e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:26:34.459160 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-ccf08d9b-fa07-47a6-87f9-e124ac61f57e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:26:34.470063 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccf08d9b-fa07-47a6-87f9-e124ac61f57e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:26:34.471633 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccf08d9b-fa07-47a6-87f9-e124ac61f57e service neutron] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:26:34.480588 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1481/2931] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:26:34 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:26:35.593058 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e21a9751-4d5c-49db-870b-45d837ae63fb service placement] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:26:35.602561 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e21a9751-4d5c-49db-870b-45d837ae63fb service placement] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:26:35.606131 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b5b02e-8fc7-4dfa-aea7-2468c77c2588 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:26:35.606717 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b5b02e-8fc7-4dfa-aea7-2468c77c2588 service neutron] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:26:35.607025 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b5b02e-8fc7-4dfa-aea7-2468c77c2588 service neutron] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:26:35.607297 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b5b02e-8fc7-4dfa-aea7-2468c77c2588 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:26:35.609004 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-07b5b02e-8fc7-4dfa-aea7-2468c77c2588 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:26:35.623093 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07b5b02e-8fc7-4dfa-aea7-2468c77c2588 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:26:35.624516 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07b5b02e-8fc7-4dfa-aea7-2468c77c2588 service neutron] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:26:35.636154 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1451/2932] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:26:35 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:26:44.476248 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=6 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:26:45.630638 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=6 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:28:01.672788 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccf08d9b-fa07-47a6-87f9-e124ac61f57e service neutron] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:28:01.688554 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccf08d9b-fa07-47a6-87f9-e124ac61f57e service neutron] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:28:01.694996 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e85cedf4-11b2-45c9-bd8e-9d27aff83db1 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '886eb5e00bbf4ec292eb962b665b37d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:28:01.695588 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e85cedf4-11b2-45c9-bd8e-9d27aff83db1 service nova] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:28:01.695868 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e85cedf4-11b2-45c9-bd8e-9d27aff83db1 service nova] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:28:01.696138 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e85cedf4-11b2-45c9-bd8e-9d27aff83db1 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:28:01.697842 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-e85cedf4-11b2-45c9-bd8e-9d27aff83db1 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:28:01.710787 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e85cedf4-11b2-45c9-bd8e-9d27aff83db1 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:28:01.712380 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e85cedf4-11b2-45c9-bd8e-9d27aff83db1 service nova] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:28:01.724294 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1482/2933] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:28:01 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 52 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:28:01.860866 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b5b02e-8fc7-4dfa-aea7-2468c77c2588 service neutron] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:28:01.869683 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b5b02e-8fc7-4dfa-aea7-2468c77c2588 service neutron] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:28:01.873818 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e248830a-7f70-4b98-8862-39a72e5a3918 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:28:01.874406 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e248830a-7f70-4b98-8862-39a72e5a3918 service neutron] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:28:01.874683 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e248830a-7f70-4b98-8862-39a72e5a3918 service neutron] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:28:01.875013 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e248830a-7f70-4b98-8862-39a72e5a3918 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:28:01.876729 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-e248830a-7f70-4b98-8862-39a72e5a3918 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:28:01.890539 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e248830a-7f70-4b98-8862-39a72e5a3918 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:28:01.891995 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e248830a-7f70-4b98-8862-39a72e5a3918 service neutron] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:28:01.903074 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1452/2934] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:28:01 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:28:03.083547 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e85cedf4-11b2-45c9-bd8e-9d27aff83db1 service nova] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:28:03.094648 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e85cedf4-11b2-45c9-bd8e-9d27aff83db1 service nova] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:28:03.098357 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b096f31-6d8e-4ac5-bf4f-b979dfd14656 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c33c45f4770d4151858e1d897d09af69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:28:03.098979 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b096f31-6d8e-4ac5-bf4f-b979dfd14656 service neutron] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:28:03.099278 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b096f31-6d8e-4ac5-bf4f-b979dfd14656 service neutron] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:28:03.099546 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b096f31-6d8e-4ac5-bf4f-b979dfd14656 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:28:03.101306 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-0b096f31-6d8e-4ac5-bf4f-b979dfd14656 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:28:03.115235 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b096f31-6d8e-4ac5-bf4f-b979dfd14656 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:28:03.116769 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b096f31-6d8e-4ac5-bf4f-b979dfd14656 service neutron] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:28:03.127640 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1483/2935] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:28:03 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:28:04.059218 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e248830a-7f70-4b98-8862-39a72e5a3918 service neutron] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:28:04.071642 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e248830a-7f70-4b98-8862-39a72e5a3918 service neutron] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:28:04.077099 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-137e463e-fa4a-4a12-b503-5559a8ffe9a6 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '886eb5e00bbf4ec292eb962b665b37d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:28:04.077709 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-137e463e-fa4a-4a12-b503-5559a8ffe9a6 service nova] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:28:04.077989 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-137e463e-fa4a-4a12-b503-5559a8ffe9a6 service nova] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:28:04.078262 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-137e463e-fa4a-4a12-b503-5559a8ffe9a6 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:28:04.079976 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-137e463e-fa4a-4a12-b503-5559a8ffe9a6 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:28:04.094900 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-137e463e-fa4a-4a12-b503-5559a8ffe9a6 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:28:04.096484 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-137e463e-fa4a-4a12-b503-5559a8ffe9a6 service nova] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:28:04.107472 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1453/2936] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:28:04 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 49 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 05:28:08.586236 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b096f31-6d8e-4ac5-bf4f-b979dfd14656 service neutron] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:28:08.599411 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b096f31-6d8e-4ac5-bf4f-b979dfd14656 service neutron] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:28:08.605415 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b8026f8-8d2b-457d-b9a7-a8b1aa1e8efb service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1972959402704b89925001cd8d90cb0e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:28:08.606117 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b8026f8-8d2b-457d-b9a7-a8b1aa1e8efb service placement] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:28:08.606275 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b8026f8-8d2b-457d-b9a7-a8b1aa1e8efb service placement] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:28:08.607469 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b8026f8-8d2b-457d-b9a7-a8b1aa1e8efb service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:28:08.609383 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-4b8026f8-8d2b-457d-b9a7-a8b1aa1e8efb service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:28:08.623658 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b8026f8-8d2b-457d-b9a7-a8b1aa1e8efb service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:28:08.625624 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b8026f8-8d2b-457d-b9a7-a8b1aa1e8efb service placement] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:28:08.639714 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1484/2937] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:28:08 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 54 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:28:11.654741 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-137e463e-fa4a-4a12-b503-5559a8ffe9a6 service nova] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:28:11.667265 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-137e463e-fa4a-4a12-b503-5559a8ffe9a6 service nova] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:28:11.672336 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45037140-73f9-4004-a433-cd1b4572484e service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1972959402704b89925001cd8d90cb0e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:28:11.672932 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45037140-73f9-4004-a433-cd1b4572484e service placement] REQUEST_METHOD: `GET` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:28:11.673209 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45037140-73f9-4004-a433-cd1b4572484e service placement] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:28:11.673476 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45037140-73f9-4004-a433-cd1b4572484e service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:28:11.675215 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-45037140-73f9-4004-a433-cd1b4572484e service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:28:11.688227 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45037140-73f9-4004-a433-cd1b4572484e service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:28:11.689808 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45037140-73f9-4004-a433-cd1b4572484e service placement] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:28:11.702446 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1454/2938] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:28:11 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 49 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:28:11.743551 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b8026f8-8d2b-457d-b9a7-a8b1aa1e8efb service placement] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:28:11.804691 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b8026f8-8d2b-457d-b9a7-a8b1aa1e8efb service placement] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:28:11.809092 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd4b1a9a-0534-47a7-a8d1-925142f67916 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:28:11.810212 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd4b1a9a-0534-47a7-a8d1-925142f67916 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:28:11.810484 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd4b1a9a-0534-47a7-a8d1-925142f67916 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:28:11.810749 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd4b1a9a-0534-47a7-a8d1-925142f67916 admin admin] PATH_INFO: `/v3/users/8f9b73d97e8145a39215f179b1398350` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:28:11.811882 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd4b1a9a-0534-47a7-a8d1-925142f67916 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8f9b73d97e8145a39215f179b1398350)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:28:11.813772 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd4b1a9a-0534-47a7-a8d1-925142f67916 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:28:11.858267 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-fd4b1a9a-0534-47a7-a8d1-925142f67916 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8f9b73d97e8145a39215f179b1398350'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:28:11.864848 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-fd4b1a9a-0534-47a7-a8d1-925142f67916 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8f9b73d97e8145a39215f179b1398350'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:28:11.877030 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-fd4b1a9a-0534-47a7-a8d1-925142f67916 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8f9b73d97e8145a39215f179b1398350'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:28:11.883741 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-fd4b1a9a-0534-47a7-a8d1-925142f67916 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8f9b73d97e8145a39215f179b1398350'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:28:11.884358 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-fd4b1a9a-0534-47a7-a8d1-925142f67916 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8f9b73d97e8145a39215f179b1398350'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:28:11.890014 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1485/2939] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:28:11 2022] DELETE /identity/v3/users/8f9b73d97e8145a39215f179b1398350 => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:28:12.060455 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45037140-73f9-4004-a433-cd1b4572484e service placement] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:28:12.071244 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45037140-73f9-4004-a433-cd1b4572484e service placement] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:28:12.127576 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20b5a094-96ab-4f4a-a52e-7caef3d22ff4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:28:12.128198 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20b5a094-96ab-4f4a-a52e-7caef3d22ff4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:28:12.128466 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20b5a094-96ab-4f4a-a52e-7caef3d22ff4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:28:12.128735 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20b5a094-96ab-4f4a-a52e-7caef3d22ff4 admin admin] PATH_INFO: `/v3/projects/05ebb49967334f8bb6f2077fce987ee3` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:28:12.129839 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20b5a094-96ab-4f4a-a52e-7caef3d22ff4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=05ebb49967334f8bb6f2077fce987ee3)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:28:12.131784 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20b5a094-96ab-4f4a-a52e-7caef3d22ff4 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:28:12.176350 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-20b5a094-96ab-4f4a-a52e-7caef3d22ff4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '05ebb49967334f8bb6f2077fce987ee3', 'request_id': 'req-20b5a094-96ab-4f4a-a52e-7caef3d22ff4'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:28:12.183186 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-20b5a094-96ab-4f4a-a52e-7caef3d22ff4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '05ebb49967334f8bb6f2077fce987ee3', 'request_id': 'req-20b5a094-96ab-4f4a-a52e-7caef3d22ff4'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:28:12.190233 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-20b5a094-96ab-4f4a-a52e-7caef3d22ff4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '05ebb49967334f8bb6f2077fce987ee3', 'request_id': 'req-20b5a094-96ab-4f4a-a52e-7caef3d22ff4'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:28:12.200260 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-20b5a094-96ab-4f4a-a52e-7caef3d22ff4 admin admin] The token cache is being invalidate because project 05ebb49967334f8bb6f2077fce987ee3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81624) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:28:12.200546 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-20b5a094-96ab-4f4a-a52e-7caef3d22ff4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:28:12.201163 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-20b5a094-96ab-4f4a-a52e-7caef3d22ff4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:28:12.202596 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1455/2940] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:28:12 2022] DELETE /identity/v3/projects/05ebb49967334f8bb6f2077fce987ee3 => generated 0 bytes in 143 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:28:12.206597 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd4b1a9a-0534-47a7-a8d1-925142f67916 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:28:12.208074 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-fd4b1a9a-0534-47a7-a8d1-925142f67916 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:28:12.221533 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd4b1a9a-0534-47a7-a8d1-925142f67916 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:28:12.259083 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:28:12.278846 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6beb6cf0-7132-4808-bdf1-6cca62a36b97 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:28:12.279481 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6beb6cf0-7132-4808-bdf1-6cca62a36b97 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:28:12.279759 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6beb6cf0-7132-4808-bdf1-6cca62a36b97 admin admin] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:28:12.280030 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6beb6cf0-7132-4808-bdf1-6cca62a36b97 admin admin] PATH_INFO: `/v3/users/86ccd1436c3d4739a3ef88f3ec94193a` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:28:12.281143 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6beb6cf0-7132-4808-bdf1-6cca62a36b97 admin admin] RBAC: Authorizing `identity:delete_user(user_id=86ccd1436c3d4739a3ef88f3ec94193a)` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:28:12.283076 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6beb6cf0-7132-4808-bdf1-6cca62a36b97 admin admin] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:28:12.327061 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-6beb6cf0-7132-4808-bdf1-6cca62a36b97 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '86ccd1436c3d4739a3ef88f3ec94193a'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:28:12.333715 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-6beb6cf0-7132-4808-bdf1-6cca62a36b97 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '86ccd1436c3d4739a3ef88f3ec94193a'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:28:12.346482 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-6beb6cf0-7132-4808-bdf1-6cca62a36b97 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '86ccd1436c3d4739a3ef88f3ec94193a'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:28:12.353270 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-6beb6cf0-7132-4808-bdf1-6cca62a36b97 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '86ccd1436c3d4739a3ef88f3ec94193a'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:28:12.354039 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.notifications [None req-6beb6cf0-7132-4808-bdf1-6cca62a36b97 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '86ccd1436c3d4739a3ef88f3ec94193a'} {{(pid=81625) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:28:12.360391 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1486/2941] 10.222.0.77 () {58 vars in 1224 bytes} [Tue Aug 30 05:28:12 2022] DELETE /identity/v3/users/86ccd1436c3d4739a3ef88f3ec94193a => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:28:12.530713 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20b5a094-96ab-4f4a-a52e-7caef3d22ff4 admin admin] Authenticating user token {{(pid=81624) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:28:12.532401 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.fernet_utils [None req-20b5a094-96ab-4f4a-a52e-7caef3d22ff4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81624) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:28:12.547294 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20b5a094-96ab-4f4a-a52e-7caef3d22ff4 admin admin] Validating token access rules against request {{(pid=81624) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:28:12.595067 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:28:12.605843 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9d03e8e-3750-4ca8-959a-c43836d6d579 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fdab92bac3f64739a0fcb5f729d40f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '46478c740f59492585d5babfa714b5b5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81624) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:28:12.606463 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9d03e8e-3750-4ca8-959a-c43836d6d579 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:28:12.606741 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9d03e8e-3750-4ca8-959a-c43836d6d579 admin admin] SCRIPT_NAME: `/identity` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:28:12.607041 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9d03e8e-3750-4ca8-959a-c43836d6d579 admin admin] PATH_INFO: `/v3/projects/067b159ec95a4432b30aa1d3fc3f98b4` {{(pid=81624) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:28:12.608142 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9d03e8e-3750-4ca8-959a-c43836d6d579 admin admin] RBAC: Authorizing `identity:delete_project(project_id=067b159ec95a4432b30aa1d3fc3f98b4)` {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:28:12.610026 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9d03e8e-3750-4ca8-959a-c43836d6d579 admin admin] RBAC: Authorization granted {{(pid=81624) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:28:12.655614 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-f9d03e8e-3750-4ca8-959a-c43836d6d579 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '067b159ec95a4432b30aa1d3fc3f98b4', 'request_id': 'req-f9d03e8e-3750-4ca8-959a-c43836d6d579'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:28:12.662622 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-f9d03e8e-3750-4ca8-959a-c43836d6d579 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '067b159ec95a4432b30aa1d3fc3f98b4', 'request_id': 'req-f9d03e8e-3750-4ca8-959a-c43836d6d579'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:28:12.668753 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-f9d03e8e-3750-4ca8-959a-c43836d6d579 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '067b159ec95a4432b30aa1d3fc3f98b4', 'request_id': 'req-f9d03e8e-3750-4ca8-959a-c43836d6d579'} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:28:12.678903 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-f9d03e8e-3750-4ca8-959a-c43836d6d579 admin admin] The token cache is being invalidate because project 067b159ec95a4432b30aa1d3fc3f98b4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81624) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 05:28:12.679188 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-f9d03e8e-3750-4ca8-959a-c43836d6d579 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:28:12.679801 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG keystone.notifications [None req-f9d03e8e-3750-4ca8-959a-c43836d6d579 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81624) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 05:28:12.681237 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: [pid: 81624|app: 0|req: 1456/2942] 10.222.0.77 () {58 vars in 1233 bytes} [Tue Aug 30 05:28:12 2022] DELETE /identity/v3/projects/067b159ec95a4432b30aa1d3fc3f98b4 => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 05:28:22.356423 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=35,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:28:22.676091 dv-cinder-853910-4-iscsi devstack@keystone.service[81624]: DEBUG dbcounter [-] [81624] Writing DB stats keystone:SELECT=28,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81624) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 05:31:09.371700 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6beb6cf0-7132-4808-bdf1-6cca62a36b97 admin admin] Authenticating user token {{(pid=81625) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 05:31:09.373142 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-6beb6cf0-7132-4808-bdf1-6cca62a36b97 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:31:09.512444 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6beb6cf0-7132-4808-bdf1-6cca62a36b97 admin admin] Validating token access rules against request {{(pid=81625) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 05:31:09.574436 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a841cb6-ec3f-4e8a-9543-8ab7f89b6416 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1972959402704b89925001cd8d90cb0e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e9919141e3b947ab8d91b55bba4a7de6', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81625) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 05:31:09.575071 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a841cb6-ec3f-4e8a-9543-8ab7f89b6416 service placement] REQUEST_METHOD: `GET` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 05:31:09.575353 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a841cb6-ec3f-4e8a-9543-8ab7f89b6416 service placement] SCRIPT_NAME: `/identity` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 05:31:09.575627 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a841cb6-ec3f-4e8a-9543-8ab7f89b6416 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81625) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 05:31:09.577381 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.fernet_utils [None req-1a841cb6-ec3f-4e8a-9543-8ab7f89b6416 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81625) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 05:31:09.589308 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a841cb6-ec3f-4e8a-9543-8ab7f89b6416 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 05:31:09.590896 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a841cb6-ec3f-4e8a-9543-8ab7f89b6416 service placement] RBAC: Authorization granted {{(pid=81625) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 05:31:09.600963 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: [pid: 81625|app: 0|req: 1487/2943] 10.222.0.77 () {60 vars in 1365 bytes} [Tue Aug 30 05:31:09 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 230 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 05:31:19.596567 dv-cinder-853910-4-iscsi devstack@keystone.service[81625]: DEBUG dbcounter [-] [81625] Writing DB stats keystone:SELECT=89 {{(pid=81625) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}